kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun MA <mj.saber1...@gmail.com>
Subject kafka 0.9 offset unknown after cleanup
Date Tue, 03 May 2016 07:55:57 GMT
Hi,

I’m using 0.9.0.1 new-consumer api. I noticed that after kafka cleans up all old log segments(reach
delete.retention time), I got unknown offset.

bin/kafka-consumer-groups.sh --bootstrap-server server:9092 --new-consumer --group testGroup
--describe
GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
testGroup, test, 0, unknown, 49, unknown, consumer-1_/10.32.241.2
testGroup, test, 1, unknown, 61, unknown, consumer-1_/10.32.241.2

In this situation, I cannot consume anything using new-consumer java driver if I disable auto-commit.
I think this happens because new-consumer driver stores offset in broker as a topic(not in
zookeeper), and after reaching delete.retention time, it got deleted and becomes unknown.
And since I disabled auto-commit, it can never know where it is, then it cannot consume anything.

Is this what happened here? What should I do in this situation?

Thanks,
Jun
Mime
View raw message