kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <...@confluent.io>
Subject Re: mirrormaker tool in 0.82beta
Date Wed, 07 Jan 2015 05:44:06 GMT
Did you set offsets.storage to kafka in the consumer of mirror maker?

Thanks,

Jun

On Mon, Jan 5, 2015 at 3:49 PM, svante karlsson <saka@csi.se> wrote:

> I'm using 0.82beta and I'm trying to push data with the mirrormaker tool
> from several remote sites to two datacenters. I'm testing this from a node
> containing zk, broker and mirrormaker and the data is pushed to a "normal"
> cluster. 3 zk and 4 brokers with replication.
>
> While the configuration seems straight forward things are a bit shaky.
>
> First, if something goes wrong on either consumer or producer side it stops
> transmitting data without dying which makes it harder to run under a
> supervisor (upstart in my case)
>
> I start pushing data to site 02 and mirrormaker pushes it to my datacenter
> - so far all good...
>
> But, when I run. (on my remote site)
> kafka-consumer-offset-checker.sh
> --group mirrormaker.from.site_02.to.site_ktv --zookeeper 192.168.0.106
>
> Could not fetch offset for [saka.test.ext_datastream,7] due to
> kafka.common.NotCoordinatorForConsumerException.
> Could not fetch offset for [saka.test.ext_datastream,4] due to
> kafka.common.NotCoordinatorForConsumerException.
> Could not fetch offset for [saka.test.ext_datastream,2] due to
> kafka.common.NotCoordinatorForConsumerException.
> Could not fetch offset for [saka.test.ext_datastream,5] due to
> kafka.common.NotCoordinatorForConsumerException.
> Could not fetch offset for [saka.test.ext_datastream,6] due to
> kafka.common.NotCoordinatorForConsumerException.
> Could not fetch offset for [saka.test.ext_datastream,3] due to
> kafka.common.NotCoordinatorForConsumerException.
> Could not fetch offset for [saka.test.ext_datastream,1] due to
> kafka.common.NotCoordinatorForConsumerException.
> Could not fetch offset for [saka.test.ext_datastream,0] due to
> kafka.common.NotCoordinatorForConsumerException.
>
> Group           Topic                          Pid Offset          logSize
>         Lag             Owner
> mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream       0
> unknown         9310560         unknown
> mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream       1
> unknown         9313497         unknown
> mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream       2
> unknown         9323623         unknown
> mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream       3
> unknown         9334005         unknown
> mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream       4
> unknown         9324176         unknown
> mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream       5
> unknown         9336504         unknown
> mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream       6
> unknown         9316139         unknown
> mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream       7
> unknown         9318770         unknown
> mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
>
>
> Since I can push data between my sites something is working but what can be
> the case for NotCoordinatorForConsumerException on a single node cluster?
>
>
> Finally the __consumer_offsets topic should probably be masked out from
> mirroring without using whitelists or blacklists
>
>
> thanks
> /svante
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message