kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stevo Slavić <ssla...@gmail.com>
Subject Re: New Consumer & committed offsets
Date Tue, 15 Sep 2015 14:05:09 GMT
Hello Damian,

Yes, there's a +1 difference. See related discussion
http://mail-archives.apache.org/mod_mbox/kafka-users/201507.mbox/%3CCAOeJiJh2SMzVn23JsoWiNk3sfsw82Jr_-kRLcNRd-oZ7pR1yWg%40mail.gmail.com%3E

Kind regards,
Stevo Slavic.

On Tue, Sep 15, 2015 at 3:56 PM, Damian Guy <damian.guy@gmail.com> wrote:

> I turned off compression and still get duplicates, but only 1 from each
> topic.
> Should the initial fetch offset for a partition be committed offset +1 ?
>
> Thanks,
> Damian
>
> On 15 September 2015 at 14:07, Damian Guy <damian.guy@gmail.com> wrote:
>
> > Hi,
> >
> > I've been trying out the new consumer and have noticed that i get
> > duplicate messages when i stop the consumer and then restart (different
> > processes, same consumer group).
> >
> > I consume all of the messages on the topic and commit the offsets for
> each
> > partition and stop the consumer. On the next run i expect to get 0
> > messages, however i get a batch of records from each partition - in this
> > case works out 1020 messages. Run it again and i get the same batch of
> > records.
> >
> > My logging shows that i've received messages with offsets lower than were
> > previously committed.
> >
> > committed:{damian_test_one-2=137669}
> > committed:{damian_test_one-0=139161}
> > committed:{damian_test_one-1=137663}
> >
> > min offsets received: {damian_test_one-0=138824,
> damian_test_one-1=137321,
> > damian_test_one-2=137331}
> >
> > I've debugged the initial fetch requests for offsets and the offsets
> match
> > up with what has been committed. Is this expected behaviour? Something to
> > do with batching of compression of message sets?
> >
> > TIA,
> > Damian
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message