kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stevo Slavić <ssla...@gmail.com>
Subject Re: New consumer - offset one gets in poll is not offset one is supposed to commit
Date Tue, 28 Jul 2015 22:16:45 GMT
Hello Jason,

Thanks for reply!

About your proposal, in general case it might be helpful. In my case it
will not help much - I'm allowing each ConsumerRecord or subset of
ConsumerRecords to be processed and ACKed independently and out of HLC
process/thread (not to block partition), and then committing largest
consecutive ACKed processed offset (+1) since current last committed offset
per partition.

Kind regards,
Stevo Slavic.

On Mon, Jul 27, 2015 at 6:52 PM, Jason Gustafson <jason@confluent.io> wrote:

> Hey Stevo,
>
> I agree that it's a little unintuitive that what you are committing is the
> next offset that should be read from and not the one that has already been
> read. We're probably constrained in that we already have a consumer which
> implements this behavior. Would it help if we added a method on
> ConsumerRecords to get the next offset (e.g. nextOffset(partition))?
>
> Thanks,
> Jason
>
> On Fri, Jul 24, 2015 at 10:11 AM, Stevo Slavić <sslavic@gmail.com> wrote:
>
> > Hello Apache Kafka community,
> >
> > Say there is only one topic with single partition and a single message on
> > it.
> > Result of calling a poll with new consumer will return ConsumerRecord for
> > that message and it will have offset of 0.
> >
> > After processing message, current KafkaConsumer implementation expects
> one
> > to commit not offset 0 as processed, but to commit offset 1 - next
> > offset/position one would like to consume.
> >
> > Does this sound strange to you as well?
> >
> > Wondering couldn't this offset+1 handling for next position to read been
> > done in one place, in KafkaConsumer implementation or broker or whatever,
> > instead of every user of KafkaConsumer having to do it.
> >
> > Kind regards,
> > Stevo Slavic.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message