kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piotr Husiatyński <...@optiopay.com>
Subject Re: Fetch API Offset
Date Tue, 21 Apr 2015 15:46:11 GMT
I'm using Go with kafka library** and sending fetch request as
described in wire protocol documentation returns messages off by one.
Are you sure that code that you wrote is not sending fetch request
with offset being one less than you pass?

** https://github.com/optiopay/kafka

On Tue, Apr 21, 2015 at 4:54 PM, Manoj Khangaonkar
<khangaonkar@gmail.com> wrote:
> Hi,
>
> I have used code like
>
> FetchRequest req = new FetchRequestBuilder()
>           .clientId(clientname)
>           .addFetch(topic, partition, offset_in_partition, 100000).build();
> FetchResponse fetchResponse = consumer.fetch(req);
>
> And it returns the message that I was looking for.
>
> The offset_in_partion is a value I get by calling
>
> offset_in_partition = messageAndOffset.nextOffset(); // call this on a
> previous message
>
> and saving it somewhere for future use.
>
> The only caveat is that fetch returns a block of messages which means I
> might
> get messages with lower offset that I just need to ignore.
>
> regards
>
>
> On Tue, Apr 21, 2015 at 6:47 AM, Piotr Husiatyński <phu@optiopay.com> wrote:
>
>> What do you mean that message has expired?
>>
>> If I will do fetch request with offset 4, I will receive first message
>> with offset 5. I think fetch is always excluding requested offset,
>> while documentation is saying it will be included in response.
>>
>> On Tue, Apr 21, 2015 at 3:35 PM, Manoj Khangaonkar
>> <khangaonkar@gmail.com> wrote:
>> > Hi,
>> >
>> > I suspect If some message from the given offset have expired, then they
>> > will not be returned.
>> >
>> > regards
>> >
>> > On Tue, Apr 21, 2015 at 5:14 AM, Piotr Husiatyński <phu@optiopay.com>
>> wrote:
>> >
>> >> According to documentation, sending fetch request with offset value
>> >> result in messages starting with given offset (including) and greater.
>> >>
>> >>
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchAPI
>> >>
>> >> > Logically one specifies the topics, partitions, and starting offset
at
>> >> which to begin the fetch and gets back a chunk of messages. In general,
>> the
>> >> return messages will have offsets larger than or equal to the starting
>> >> offset
>> >>
>> >> I'm sending fetch request with offset 5 and first message I'm getting
>> >> is offset 6. Am I doing something wrong or documentation has to be
>> >> fixed?
>> >>
>> >
>> >
>> >
>> > --
>> > http://khangaonkar.blogspot.com/
>>
>
>
>
> --
> http://khangaonkar.blogspot.com/

Mime
View raw message