kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ye Hong <Ye.H...@audiencescience.com>
Subject Resetting consumer offsets after moving to offset.storage=kafka
Date Wed, 09 Sep 2015 21:17:52 GMT

We have a consumer that under certain circumstances may lose data. To guard against such data
loss, we have a tool that periodically pulls and stores offsets from zk. Once a data loss
takes place, we use our historical offsets to reset the consumer offset on zk.
With offset.storage=zookeeper, the tool just simply calls kafka-run-class.sh kafka.tools.ExportZkOffsets/ImportZkOffsets.
However, after moving to offset.storage=kafka, we can no longer call ExportZkOffsets/ImportZkOffsets.
For offset export, I suppose we can call the REST API of Burrow to get the same results. However,
I couldn't find an easy way to reset offsets that’s comparable to ImportZkOffsets. Could
someone shed some lights on what we should do?

View raw message