kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Gustafson <ja...@confluent.io>
Subject Re: SimpleConsumer.getOffsetsBefore() in 0.9 KafkaConsumer
Date Fri, 22 Jan 2016 19:37:44 GMT
The offset API is one of the known gaps in the new consumer. There is a
JIRA (KAFKA-1332), but we might need a KIP to make that change now that the
API is released. For now, Gwen's suggestion is the only way to do it.

-Jason

On Thu, Jan 21, 2016 at 8:22 PM, Gwen Shapira <gwen@confluent.io> wrote:

> Hi Robert!
>
> Jason is the expert, and I hope he'll respond soon.
>
> Meanwhile: I think that you can do what you are trying to do by:
> 1. call position() to get the current position you are consuming
> 2. call seekToEnd() and then position(), which will give you the last
> position at the point in which you called seekToEnd()
> 3. Use another seek() call to go to the offset you really want to get to.
>
> I don't think there are current plans to add getOffsetsBefore, but maybe we
> need it for the use-case you specified.
> I think the developer mailing list (or a JIRA) will be a better place for
> an API discussion.
>
> Gwen
>
> On Wed, Jan 20, 2016 at 3:09 AM, Robert Metzger <rmetzger@apache.org>
> wrote:
>
> > Hi,
> >
> > I'm currently looking into implementing a load shedding strategy into
> > Flink's Kafka consumer.
> >
> > Therefore, I would like to allow users to request the latest offset of
> the
> > subscribed TopicPartitions, so that they can
> > a) determine the lag
> > b) maybe set the next fetch offset to the latest offset (or anything in
> > between)
> >
> > With the SimpleConsumer, there is the getOffsetsBefore() method, which
> > allows to request the largest offset.
> > For the new KafkaConsumer, I only found the seekToEnd() and
> > seekToBeginning() methods, but they don't allow to get the actual end or
> > beginning.
> >
> >
> > Are there plans to add such an API to the KafkaConsumer again?
> > Can you suggest any workarounds?
> >
> > Regards,
> > Robert
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message