kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrei <faithlessfri...@gmail.com>
Subject Re: Fetch API Offset
Date Tue, 21 Apr 2015 19:02:18 GMT
@Piotr I can confirm that in native (Java/Scala) library returned message
offsets start with the one specified in request, not the next one. Note
sure about underlying protocol, though.

As long as you also know Python, I would suggest you looking at
kafka-python [1] library that is pretty readable and has already solved
most issues you will encounter.

[1]: https://github.com/mumrah/kafka-python



On Tue, Apr 21, 2015 at 6:46 PM, Piotr Husiatyński <phu@optiopay.com> wrote:

> I'm using Go with kafka library** and sending fetch request as
> described in wire protocol documentation returns messages off by one.
> Are you sure that code that you wrote is not sending fetch request
> with offset being one less than you pass?
>
> ** https://github.com/optiopay/kafka
>
> On Tue, Apr 21, 2015 at 4:54 PM, Manoj Khangaonkar
> <khangaonkar@gmail.com> wrote:
> > Hi,
> >
> > I have used code like
> >
> > FetchRequest req = new FetchRequestBuilder()
> >           .clientId(clientname)
> >           .addFetch(topic, partition, offset_in_partition,
> 100000).build();
> > FetchResponse fetchResponse = consumer.fetch(req);
> >
> > And it returns the message that I was looking for.
> >
> > The offset_in_partion is a value I get by calling
> >
> > offset_in_partition = messageAndOffset.nextOffset(); // call this on a
> > previous message
> >
> > and saving it somewhere for future use.
> >
> > The only caveat is that fetch returns a block of messages which means I
> > might
> > get messages with lower offset that I just need to ignore.
> >
> > regards
> >
> >
> > On Tue, Apr 21, 2015 at 6:47 AM, Piotr Husiatyński <phu@optiopay.com>
> wrote:
> >
> >> What do you mean that message has expired?
> >>
> >> If I will do fetch request with offset 4, I will receive first message
> >> with offset 5. I think fetch is always excluding requested offset,
> >> while documentation is saying it will be included in response.
> >>
> >> On Tue, Apr 21, 2015 at 3:35 PM, Manoj Khangaonkar
> >> <khangaonkar@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > I suspect If some message from the given offset have expired, then
> they
> >> > will not be returned.
> >> >
> >> > regards
> >> >
> >> > On Tue, Apr 21, 2015 at 5:14 AM, Piotr Husiatyński <phu@optiopay.com>
> >> wrote:
> >> >
> >> >> According to documentation, sending fetch request with offset value
> >> >> result in messages starting with given offset (including) and
> greater.
> >> >>
> >> >>
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchAPI
> >> >>
> >> >> > Logically one specifies the topics, partitions, and starting
> offset at
> >> >> which to begin the fetch and gets back a chunk of messages. In
> general,
> >> the
> >> >> return messages will have offsets larger than or equal to the
> starting
> >> >> offset
> >> >>
> >> >> I'm sending fetch request with offset 5 and first message I'm getting
> >> >> is offset 6. Am I doing something wrong or documentation has to be
> >> >> fixed?
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > http://khangaonkar.blogspot.com/
> >>
> >
> >
> >
> > --
> > http://khangaonkar.blogspot.com/
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message