samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@linkedin.com>
Subject Re: Problem testing creation of ConsumerConfig object (re SAMZA-77)
Date Tue, 19 Nov 2013 01:06:36 GMT
Hey Gary,

Wow, great! :)

Kafka, does indeed have some requirements on what config parameters it
needs, and it does verify these at instantiation time. The parameters
you'll need to set are:

  
systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactor
y
  systems.kafka.consumer.zookeeper.connect=localhost:2181/
  systems.kafka.producer.metadata.broker.list=localhost:9092


Note that, in this case, the system is called "kafka". In your example,
you'd use the "systems.appname.*" prefix (or change your test to call
getKafkaSystemConsumerConfig("kafka") ).

Kafka does come with a way to do this in their TestUtils class:

  
https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/u
tils/TestUtils.scala?source=c

The method you're probably looking for is:

  createConsumerProperties

And:

  getProducerConfig

What I'd recommend is just creating a java.util.Properties instance, and
adding all appropriate configs to it (either with putAll, or put). Then
I'd use the MapConfig class to convert the Properties to a Config object:

  val config = new MapConfig(myProps);

This is a bit easier than going the factory route that you're on right
now, and it can all be done inside your unit test, instead of having extra
properties files.

Cheers,
Chris

On 11/18/13 4:53 PM, "Garry Turkington" <g.turkington@improvedigital.com>
wrote:

>Hi,
>
>I thought SAMZA-77 looked like fun so made the changes required to add
>the stringified UUID to the client and group IDs.  All nice and easy.
>Then I tried adding some test cases and think I'm managing to make this
>more difficult than it needs to be. :)
>
>In my test case I have the following, which I believe is the right chain
>of calls to create a KafkaConfig object (I created a simple config file
>in the resources directory):
>
>  val factory = new PropertiesConfigFactory()
>    val config = 
>factory.getConfig(URI.create("file://%s/src/test/resources/test.properties
><file:///\\%25s\src\test\resources\test.properties>" format new File
>(".").getCanonicalPath))
>val kafkaConfig = new KafkaConfig(config)
>
>All well and good but then I try and call the appropriate method on the
>KafkaConfig object (with some uncertainty of what the argument needs map
>to):
>
>val config1 = kafkaConfig.getKafkaSystemConsumerConfig("appname")
>
>This causes an exception to be thrown, the highlights being:
>
>java.lang.IllegalArgumentException: requirement failed: Missing required
>property 'zookeeper.connect'
>        at scala.Predef$.require(Predef.scala:145)
>        at 
>kafka.utils.VerifiableProperties.getString(VerifiableProperties.scala:175)
>        at kafka.utils.ZKConfig.<init>(ZkUtils.scala:775)
>        at kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:73)
>        at kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:77)
>        at 
>org.apache.samza.config.KafkaConfig.getKafkaSystemConsumerConfig(KafkaConf
>ig.scala:91)
>
>This suggests that to create a ConsumerConfig requires the whole ZK/Kafka
>stack to be instantiated - is this the case and if so pointers to the
>minimum config set required to get it working?  I tried to piece
>something together from the other test cases but couldn't find the right
>magic.
>
>Or is there an easier way to do all this? :)
>
>Thanks
>Garry


Mime
View raw message