samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <>
Subject Re: Problem testing creation of ConsumerConfig object (re SAMZA-77)
Date Tue, 19 Nov 2013 01:06:36 GMT
Hey Gary,

Wow, great! :)

Kafka, does indeed have some requirements on what config parameters it
needs, and it does verify these at instantiation time. The parameters
you'll need to set are:


Note that, in this case, the system is called "kafka". In your example,
you'd use the "systems.appname.*" prefix (or change your test to call
getKafkaSystemConsumerConfig("kafka") ).

Kafka does come with a way to do this in their TestUtils class:

The method you're probably looking for is:




What I'd recommend is just creating a java.util.Properties instance, and
adding all appropriate configs to it (either with putAll, or put). Then
I'd use the MapConfig class to convert the Properties to a Config object:

  val config = new MapConfig(myProps);

This is a bit easier than going the factory route that you're on right
now, and it can all be done inside your unit test, instead of having extra
properties files.


On 11/18/13 4:53 PM, "Garry Turkington" <>

>I thought SAMZA-77 looked like fun so made the changes required to add
>the stringified UUID to the client and group IDs.  All nice and easy.
>Then I tried adding some test cases and think I'm managing to make this
>more difficult than it needs to be. :)
>In my test case I have the following, which I believe is the right chain
>of calls to create a KafkaConfig object (I created a simple config file
>in the resources directory):
>  val factory = new PropertiesConfigFactory()
>    val config = 
><file:///\\%25s\src\test\resources\>" format new File
>val kafkaConfig = new KafkaConfig(config)
>All well and good but then I try and call the appropriate method on the
>KafkaConfig object (with some uncertainty of what the argument needs map
>val config1 = kafkaConfig.getKafkaSystemConsumerConfig("appname")
>This causes an exception to be thrown, the highlights being:
>java.lang.IllegalArgumentException: requirement failed: Missing required
>property 'zookeeper.connect'
>        at scala.Predef$.require(Predef.scala:145)
>        at 
>        at kafka.utils.ZKConfig.<init>(ZkUtils.scala:775)
>        at kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:73)
>        at kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:77)
>        at 
>This suggests that to create a ConsumerConfig requires the whole ZK/Kafka
>stack to be instantiated - is this the case and if so pointers to the
>minimum config set required to get it working?  I tried to piece
>something together from the other test cases but couldn't find the right
>Or is there an easier way to do all this? :)

View raw message