samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Garry Turkington <>
Subject RE: Problem testing creation of ConsumerConfig object (re SAMZA-77)
Date Wed, 20 Nov 2013 01:34:49 GMT
Hi Chris,

Thanks for the pointers, very useful.  I got my tests to pass but seem to have broken the
TopicMetadataCache in the process; I assume somewhere internally it's using the client id
(not group id) as part of an equality test.  But I'll dig into it tomorrow.


-----Original Message-----
From: Chris Riccomini [] 
Sent: 19 November 2013 01:07
Subject: Re: Problem testing creation of ConsumerConfig object (re SAMZA-77)

Hey Gary,

Wow, great! :)

Kafka, does indeed have some requirements on what config parameters it needs, and it does
verify these at instantiation time. The parameters you'll need to set are:


Note that, in this case, the system is called "kafka". In your example, you'd use the "systems.appname.*"
prefix (or change your test to call
getKafkaSystemConsumerConfig("kafka") ).

Kafka does come with a way to do this in their TestUtils class:

The method you're probably looking for is:




What I'd recommend is just creating a java.util.Properties instance, and adding all appropriate
configs to it (either with putAll, or put). Then I'd use the MapConfig class to convert the
Properties to a Config object:

  val config = new MapConfig(myProps);

This is a bit easier than going the factory route that you're on right now, and it can all
be done inside your unit test, instead of having extra properties files.


On 11/18/13 4:53 PM, "Garry Turkington" <>

>I thought SAMZA-77 looked like fun so made the changes required to add 
>the stringified UUID to the client and group IDs.  All nice and easy.
>Then I tried adding some test cases and think I'm managing to make this 
>more difficult than it needs to be. :)
>In my test case I have the following, which I believe is the right 
>chain of calls to create a KafkaConfig object (I created a simple 
>config file in the resources directory):
>  val factory = new PropertiesConfigFactory()
>    val config =
>ies <file:///\\%25s\src\test\resources\>" format new 
>val kafkaConfig = new KafkaConfig(config)
>All well and good but then I try and call the appropriate method on the 
>KafkaConfig object (with some uncertainty of what the argument needs 
>val config1 = kafkaConfig.getKafkaSystemConsumerConfig("appname")
>This causes an exception to be thrown, the highlights being:
>java.lang.IllegalArgumentException: requirement failed: Missing 
>required property 'zookeeper.connect'
>        at scala.Predef$.require(Predef.scala:145)
>        at
>        at kafka.utils.ZKConfig.<init>(ZkUtils.scala:775)
>        at kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:73)
>        at kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:77)
>        at
>This suggests that to create a ConsumerConfig requires the whole 
>ZK/Kafka stack to be instantiated - is this the case and if so pointers 
>to the minimum config set required to get it working?  I tried to piece 
>something together from the other test cases but couldn't find the 
>right magic.
>Or is there an easier way to do all this? :)

No virus found in this message.
Checked by AVG -
Version: 2013.0.3426 / Virus Database: 3629/6841 - Release Date: 11/16/13

View raw message