kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Helleren, Erik" <Erik.Helle...@cmegroup.com>
Subject Re: Resetting consumer offsets after moving to offset.storage=kafka
Date Wed, 09 Sep 2015 21:34:08 GMT
It is possible to commit offsets using the SimpleConsumer API to kafka or
zookeeper for any GroupID, topic, and partition tuple.  There are some
difficulties with the SimpleConsumer, but it should be able to make the
call within your app.  See the scala Doc here:
a.javaapi.consumer.SimpleConsumer And look for the commitOffsets function.

I am curious, in what situations are there data loss?

On 9/9/15, 4:17 PM, "Ye Hong" <Ye.Hong@audiencescience.com> wrote:

>We have a consumer that under certain circumstances may lose data. To
>guard against such data loss, we have a tool that periodically pulls and
>stores offsets from zk. Once a data loss takes place, we use our
>historical offsets to reset the consumer offset on zk.
>With offset.storage=zookeeper, the tool just simply calls
>kafka-run-class.sh kafka.tools.ExportZkOffsets/ImportZkOffsets. However,
>after moving to offset.storage=kafka, we can no longer call
>For offset export, I suppose we can call the REST API of Burrow to get
>the same results. However, I couldn't find an easy way to reset offsets
>that¹s comparable to ImportZkOffsets. Could someone shed some lights on
>what we should do?

View raw message