kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: MirrorMaker consumer does not use broker.list property
Date Wed, 29 May 2013 04:05:10 GMT
Consumer only needs zk.connect, not broker.list.

Thanks,

Jun


On Tue, May 28, 2013 at 11:31 AM, Aman Bazayev <abazayev@talenttech.com>wrote:

> Hi,
>
> I'd like to bump this issue:
>
> https://mail-archives.apache.org/mod_mbox/kafka-users/201212.mbox/%3CFA0E8A0482D176408729D604142248F319D22CA7%40EXCHANGE14.actuate.com%3E
> as I'm encountering the same problem.
>
> It seems that the MirrorMaker does the following:
> 1) if zk.connect is defined for the consumer, MirrorMaker script ignores
> broker.list value
> 2) if zk.connect is not definied for the consumer, MirrorMaker fails with
> an error message:
>
> [2013-05-28 11:24:57,457] INFO group1_ip-<hostname> Connecting to zookeeper
> instance at null (kafka.consumer.ZookeeperConsumerConnector)
> [2013-05-28 11:24:57,457] INFO Initiating client connection,
> connectString=null sessionTimeout=6000
> watcher=org.I0Itec.zkclient.ZkClient@c820344
> (org.apache.zookeeper.ZooKeeper)
> [2013-05-28 11:24:57,461] INFO Starting ZkClient event thread.
> (org.I0Itec.zkclient.ZkEventThread)
> Exception in thread "main" java.lang.NullPointerException
>     at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:361)
>     at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:332)
>     at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:383)
>     at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:64)
>     at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:872)
>     at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
>     at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
>     at
>
> kafka.consumer.ZookeeperConsumerConnector.connectZk(ZookeeperConsumerConnector.scala:152)
>     at
>
> kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:122)
>     at
>
> kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:129)
>     at kafka.tools.MirrorMaker$$anonfun$3.apply(MirrorMaker.scala:102)
>     at kafka.tools.MirrorMaker$$anonfun$3.apply(MirrorMaker.scala:102)
>     at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
>     at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
>     at
>
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
>     at scala.collection.immutable.List.foreach(List.scala:45)
>     at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
>     at scala.collection.immutable.List.map(List.scala:45)
>     at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:102)
>     at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
>
> The command I run is
>
> $KAFKA_ROOT/bin/kafka-run-class.sh kafka.tools.MirrorMaker
> --consumer.config ./mirror_consumer.properties --producer.config
> ./mirror_producer.properties --whitelist=".*" --num.streams 1
>
> The ./mirror_consumer.properties file contains:
>
> broker.list=1:localhost:59092,2:localhost:59093
> groupid=group1
> shallowiterator.enable=true
>
> When broker.list is commented out and zk.connect is defined instead (or
> when both are defined), mirroring happens successfully.
>
> Any leads?
>
> Thank you,
> Aman
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message