kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damian Guy <damian....@gmail.com>
Subject Re: New Consumer & committed offsets
Date Tue, 15 Sep 2015 13:56:59 GMT
I turned off compression and still get duplicates, but only 1 from each
topic.
Should the initial fetch offset for a partition be committed offset +1 ?

Thanks,
Damian

On 15 September 2015 at 14:07, Damian Guy <damian.guy@gmail.com> wrote:

> Hi,
>
> I've been trying out the new consumer and have noticed that i get
> duplicate messages when i stop the consumer and then restart (different
> processes, same consumer group).
>
> I consume all of the messages on the topic and commit the offsets for each
> partition and stop the consumer. On the next run i expect to get 0
> messages, however i get a batch of records from each partition - in this
> case works out 1020 messages. Run it again and i get the same batch of
> records.
>
> My logging shows that i've received messages with offsets lower than were
> previously committed.
>
> committed:{damian_test_one-2=137669}
> committed:{damian_test_one-0=139161}
> committed:{damian_test_one-1=137663}
>
> min offsets received: {damian_test_one-0=138824, damian_test_one-1=137321,
> damian_test_one-2=137331}
>
> I've debugged the initial fetch requests for offsets and the offsets match
> up with what has been committed. Is this expected behaviour? Something to
> do with batching of compression of message sets?
>
> TIA,
> Damian
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message