samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Garry Turkington <g.turking...@improvedigital.com>
Subject RE: Problem testing creation of ConsumerConfig object (re SAMZA-77)
Date Wed, 20 Nov 2013 01:34:49 GMT
Hi Chris,

Thanks for the pointers, very useful.  I got my tests to pass but seem to have broken the
TopicMetadataCache in the process; I assume somewhere internally it's using the client id
(not group id) as part of an equality test.  But I'll dig into it tomorrow.

Thanks,
Garry

-----Original Message-----
From: Chris Riccomini [mailto:criccomini@linkedin.com] 
Sent: 19 November 2013 01:07
To: dev@samza.incubator.apache.org
Subject: Re: Problem testing creation of ConsumerConfig object (re SAMZA-77)

Hey Gary,

Wow, great! :)

Kafka, does indeed have some requirements on what config parameters it needs, and it does
verify these at instantiation time. The parameters you'll need to set are:

  
systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactor
y
  systems.kafka.consumer.zookeeper.connect=localhost:2181/
  systems.kafka.producer.metadata.broker.list=localhost:9092


Note that, in this case, the system is called "kafka". In your example, you'd use the "systems.appname.*"
prefix (or change your test to call
getKafkaSystemConsumerConfig("kafka") ).

Kafka does come with a way to do this in their TestUtils class:

  
https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/u
tils/TestUtils.scala?source=c

The method you're probably looking for is:

  createConsumerProperties

And:

  getProducerConfig

What I'd recommend is just creating a java.util.Properties instance, and adding all appropriate
configs to it (either with putAll, or put). Then I'd use the MapConfig class to convert the
Properties to a Config object:

  val config = new MapConfig(myProps);

This is a bit easier than going the factory route that you're on right now, and it can all
be done inside your unit test, instead of having extra properties files.

Cheers,
Chris

On 11/18/13 4:53 PM, "Garry Turkington" <g.turkington@improvedigital.com>
wrote:

>Hi,
>
>I thought SAMZA-77 looked like fun so made the changes required to add 
>the stringified UUID to the client and group IDs.  All nice and easy.
>Then I tried adding some test cases and think I'm managing to make this 
>more difficult than it needs to be. :)
>
>In my test case I have the following, which I believe is the right 
>chain of calls to create a KafkaConfig object (I created a simple 
>config file in the resources directory):
>
>  val factory = new PropertiesConfigFactory()
>    val config =
>factory.getConfig(URI.create("file://%s/src/test/resources/test.propert
>ies <file:///\\%25s\src\test\resources\test.properties>" format new 
>File
>(".").getCanonicalPath))
>val kafkaConfig = new KafkaConfig(config)
>
>All well and good but then I try and call the appropriate method on the 
>KafkaConfig object (with some uncertainty of what the argument needs 
>map
>to):
>
>val config1 = kafkaConfig.getKafkaSystemConsumerConfig("appname")
>
>This causes an exception to be thrown, the highlights being:
>
>java.lang.IllegalArgumentException: requirement failed: Missing 
>required property 'zookeeper.connect'
>        at scala.Predef$.require(Predef.scala:145)
>        at
>kafka.utils.VerifiableProperties.getString(VerifiableProperties.scala:175)
>        at kafka.utils.ZKConfig.<init>(ZkUtils.scala:775)
>        at kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:73)
>        at kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:77)
>        at
>org.apache.samza.config.KafkaConfig.getKafkaSystemConsumerConfig(KafkaC
>onf
>ig.scala:91)
>
>This suggests that to create a ConsumerConfig requires the whole 
>ZK/Kafka stack to be instantiated - is this the case and if so pointers 
>to the minimum config set required to get it working?  I tried to piece 
>something together from the other test cases but couldn't find the 
>right magic.
>
>Or is there an easier way to do all this? :)
>
>Thanks
>Garry


-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2013.0.3426 / Virus Database: 3629/6841 - Release Date: 11/16/13

Mime
View raw message