kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aman Bazayev <abaza...@talenttech.com>
Subject Re: MirrorMaker consumer does not use broker.list property
Date Wed, 29 May 2013 18:25:42 GMT
Our design goal is to have two geographically distributed sets of Kafka in
sync. Since the data travels across the Internet, it needs to be
encrypted-in-transit.

We are currently looking into using MirrorMaker and I was hoping to utilize
SSL tunnels (implemented via stunnel) as a solution. The idea is that
source ZooKeeper reports a list of source Kafka's, stunnel sets up SSL
tunnels and consumer then gets pointed at local endpoints of the tunnels.

If you have ideas (better yet, solutions that have been used in production)
on how to achieve the goal described in the first paragraph, I'm all ears.

Regards,
Aman

On Wed, May 29, 2013 at 8:22 AM, Jun Rao <junrao@gmail.com> wrote:

> No, MirrorMaker uses our high level consumer which ignores broker.list. Is
> there a particular reason that you want the consumer in MirrorMaker to know
> the broker list?
>
> Thanks,
>
> Jun
>
>
> On Tue, May 28, 2013 at 11:24 PM, Aman Bazayev <abazayev@talenttech.com
> >wrote:
>
> > Thanks for the response. What if I'm trying to set a broker.list for
> > consumer statically, is there a way to do that?
> >
> > On Tue, May 28, 2013 at 9:05 PM, Jun Rao <junrao@gmail.com> wrote:
> >
> > > Consumer only needs zk.connect, not broker.list.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Tue, May 28, 2013 at 11:31 AM, Aman Bazayev <
> abazayev@talenttech.com
> > > >wrote:
> > >
> > > > Hi,
> > > >
> > > > I'd like to bump this issue:
> > > >
> > > >
> > >
> >
> https://mail-archives.apache.org/mod_mbox/kafka-users/201212.mbox/%3CFA0E8A0482D176408729D604142248F319D22CA7%40EXCHANGE14.actuate.com%3E
> > > > as I'm encountering the same problem.
> > > >
> > > > It seems that the MirrorMaker does the following:
> > > > 1) if zk.connect is defined for the consumer, MirrorMaker script
> > ignores
> > > > broker.list value
> > > > 2) if zk.connect is not definied for the consumer, MirrorMaker fails
> > with
> > > > an error message:
> > > >
> > > > [2013-05-28 11:24:57,457] INFO group1_ip-<hostname> Connecting to
> > > zookeeper
> > > > instance at null (kafka.consumer.ZookeeperConsumerConnector)
> > > > [2013-05-28 11:24:57,457] INFO Initiating client connection,
> > > > connectString=null sessionTimeout=6000
> > > > watcher=org.I0Itec.zkclient.ZkClient@c820344
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2013-05-28 11:24:57,461] INFO Starting ZkClient event thread.
> > > > (org.I0Itec.zkclient.ZkEventThread)
> > > > Exception in thread "main" java.lang.NullPointerException
> > > >     at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:361)
> > > >     at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:332)
> > > >     at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:383)
> > > >     at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:64)
> > > >     at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:872)
> > > >     at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
> > > >     at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.connectZk(ZookeeperConsumerConnector.scala:152)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:122)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:129)
> > > >     at
> kafka.tools.MirrorMaker$$anonfun$3.apply(MirrorMaker.scala:102)
> > > >     at
> kafka.tools.MirrorMaker$$anonfun$3.apply(MirrorMaker.scala:102)
> > > >     at
> > > >
> > > >
> > >
> >
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> > > >     at
> > > >
> > > >
> > >
> >
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> > > >     at
> > > >
> > > >
> > >
> >
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
> > > >     at scala.collection.immutable.List.foreach(List.scala:45)
> > > >     at
> > > > scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
> > > >     at scala.collection.immutable.List.map(List.scala:45)
> > > >     at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:102)
> > > >     at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> > > >
> > > > The command I run is
> > > >
> > > > $KAFKA_ROOT/bin/kafka-run-class.sh kafka.tools.MirrorMaker
> > > > --consumer.config ./mirror_consumer.properties --producer.config
> > > > ./mirror_producer.properties --whitelist=".*" --num.streams 1
> > > >
> > > > The ./mirror_consumer.properties file contains:
> > > >
> > > > broker.list=1:localhost:59092,2:localhost:59093
> > > > groupid=group1
> > > > shallowiterator.enable=true
> > > >
> > > > When broker.list is commented out and zk.connect is defined instead
> (or
> > > > when both are defined), mirroring happens successfully.
> > > >
> > > > Any leads?
> > > >
> > > > Thank you,
> > > > Aman
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message