kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerard Klijs <gerard.kl...@dizzit.com>
Subject Re: kafka 0.9 offset unknown after cleanup
Date Tue, 03 May 2016 08:15:45 GMT
Looks like it, you need to be sure the offset topic is using compaction,
and the broker is set to enable compaction.

On Tue, May 3, 2016 at 9:56 AM Jun MA <mj.saber1990@gmail.com> wrote:

> Hi,
>
> I’m using 0.9.0.1 new-consumer api. I noticed that after kafka cleans up
> all old log segments(reach delete.retention time), I got unknown offset.
>
> bin/kafka-consumer-groups.sh --bootstrap-server server:9092 --new-consumer
> --group testGroup --describe
> GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
> testGroup, test, 0, unknown, 49, unknown, consumer-1_/10.32.241.2
> testGroup, test, 1, unknown, 61, unknown, consumer-1_/10.32.241.2
>
> In this situation, I cannot consume anything using new-consumer java
> driver if I disable auto-commit.
> I think this happens because new-consumer driver stores offset in broker
> as a topic(not in zookeeper), and after reaching delete.retention time, it
> got deleted and becomes unknown. And since I disabled auto-commit, it can
> never know where it is, then it cannot consume anything.
>
> Is this what happened here? What should I do in this situation?
>
> Thanks,
> Jun

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message