kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kris K <squareksc...@gmail.com>
Subject Re: offset storage as kafka with zookeeper 3.4.6
Date Thu, 11 Jun 2015 15:53:15 GMT
>If you want to "move" offsets from zookeeper to Kafka then yes you
>need to have a phase where all consumers in your group set dual commit
>to true. If you are starting a fresh consumer group then you can
>turn off dual-commit.
I followed these steps to move the offsets from zookeeper to kafka:
1. Set dual commit to true, exclude.internal.topics=false and offset
storage to kafka on all 3 consumer.properties files
2. Rolling restart on all 3 brokers. All the consumers are high level
consumers with auto commit enable set to false
3. Left these settings for about an hour while data kept flowing through
some topics (not all)
4. Used the utility ./kafka-console-consumer.sh --topic __consumer_offsets
--zookeeper xxx:2181,yyy:2181,zzz:2181 --formatter
"kafka.server.OffsetManager\$OffsetsMessageFormatter" --consumer.config
../config/consumer.properties and found that nothing is written to this
topic
5. Changed dual commit to false followed by a rolling restart. All the
consumers are high level consumers
6. Zookeeper offsets kept changing while nothing gets written to
__consumer_offsets

In order to reproduce the issue:
1. Brought down all 3 brokers and 3 nodes of zookeeper
2. Deleted all the contents of snapshot and transaction log directories of
zookeeper
3. Added myid files in snapshot directories on all zk nodes with node ids
4. Deleted all the contents of kafka log directories on all 3 brokers
5. Set dual commit to false, exclude.internal.topics=false and offset
storage to kafka on all 3 consumer.properties files
6. Brought the environment up. All the consumers are high level consumers
with auto commit enable set to false
7. Consumer offsets still got recorded on zookeeper and kept changing
while __consumer_offsets
was empty

When I did a standalone installation with single broker and used the
zookeeper that comes with kafka, the offsets got written to __consumer_offsets.
This made me ask the question about using zookeeper 3.4.6 against the one
the comes with kafka.

>You can
>also check consumer mbeans that give the KafkaCommitRate or enable
>trace logging in either the consumer or the broker's request logs to
>check if offset commit request are getting sent out to the cluster.
I will check on this

Thanks,
Kris


On Thu, Jun 11, 2015 at 7:45 AM, Joel Koshy <jjkoshy.w@gmail.com> wrote:

> > Is it mandatory to use the zookeeper that comes with kafka for offset
> > storage to be migrated to kafka?
> If you want to "move" offsets from zookeeper to Kafka then yes you
> need to have a phase where all consumers in your group set dual commit
> to true. If you are starting a fresh consumer group then you can
> turn off dual-commit.
>
> > But nothing is being written to this topic, while the consumer offsets
> > continue to reside on zookeeper.
>
> The zookeeper offsets won't be removed. However, are they changing?
> How are you verifying that nothing is written to this topic? If you
> are trying to consume it, then you will need to set
> exclude.internal.topics=false in your consumer properties. You can
> also check consumer mbeans that give the KafkaCommitRate or enable
> trace logging in either the consumer or the broker's request logs to
> check if offset commit request are getting sent out to the cluster.
>
> On Thu, Jun 11, 2015 at 01:03:09AM -0700, Kris K wrote:
> > I am trying to migrate the offset storage to kafka (3 brokers of version
> > 0.8.2.1) using the consumer property offsets.storage=kafka.  I noticed
> that
> > a new topic, __consumer_offsets got created.
> > But nothing is being written to this topic, while the consumer offsets
> > continue to reside on zookeeper.
> >
> > I am using a 3 node zookeeper ensemble (version 3.4.6) and not using the
> > one that comes with kafka.
> >
> > The current config consumer.properties now contains:
> >
> > offsets.storage=kafka
> > dual.commit.enabled=false
> > exclude.internal.topics=false
> >
> > Is it mandatory to use the zookeeper that comes with kafka for offset
> > storage to be migrated to kafka?
> >
> > I tried both the approaches:
> >
> > 1. As listed on slide 34 of
> > http://www.slideshare.net/jjkoshy/offset-management-in-kafka.
> > 2. By deleting the zookeeper data directories and kafka log directories.
> >
> > None of them worked.
> >
> > Thanks
> > Kris
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message