kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jiangjie Qin <j...@linkedin.com.INVALID>
Subject Re: Best way to erase consumer state
Date Fri, 05 Dec 2014 08:23:07 GMT
One thing might worth checking is the offset.storage value in your
consumer config. Is it set to ³kafka² or ³zookeeper² for you consumer? If
the offset is set to kafka, then the offset will be stored in the offset
topic in Kafka instead of in zookeeper. So deleting zookeeper path won¹t
work for those consumers.
Actually the offsets for those consumers cannot be easily removed so far.
KAFKA-1784 is aiming to provide a library to make it easier.

‹Jiangjie (Becket) Qin

On 12/4/14, 10:55 PM, "Ben Drees" <ben.drees@gmail.com> wrote:

>What is the best way to erase a stopped, multiprocess consumer's state so
>that it can be reset (to "smallest offsets", for example) when the
>is restarted?  Deleting /consumers/<group-name> from Zookeeper has had the
>desired effect in the past, but I ran into a situation today where some of
>the consumer processes jumped to the desired offsets (in this case
>"largest") on restart but others did not.  I'm not sure whether the issue
>was replication lag across the Zookeeper cluster or cached state in Kafka.
>I¹m using kafka-python.

View raw message