kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stevo Slavić <ssla...@gmail.com>
Subject Re: Blocked consumer - next offset same as current offset
Date Wed, 11 Nov 2015 11:15:23 GMT
Not sure if related, but this line in AbstractFetcherThread (Kafka 0.8.2
branch) looks suspicious

This logic seems to have been changed significantly via
for https://issues.apache.org/jira/browse/KAFKA-1461
so in Kafka 0.9 branch it looks like this

Have dirty workaround for the moment, not trusting nextOffset info from
MessageAndOffset, transforming response to a new one with custom logic
calculating nextOffset (= offset used in fetch request + index of message
in message set). Will recheck at latest once 0.9 is fully released.

Unwanted behavior seems would get triggered only when consumer lag is of
appropriate size - I guess async producer made this issue more likely,
because of batch sending multiple messages to same partition.

On Wed, Nov 11, 2015 at 9:40 AM, Stevo Slavić <sslavic@gmail.com> wrote:

> Hello Apache Kafka community,
> I'm using simple consumer with Kafka and noticed that under some
> conditions fetch response message set for a partition can contain at least
> one (if not all) MessageAndOffset with nextOffset being equal to current
> (committed) offset, offset used in fetch request. Not sure how it's related
> but I could notice this behavior especially often when I was using new
> async producer, and when fetch request was able to fetch several messages
> all the way to the end of the partition.
> Is this a feature or a bug?
> Kind regards,
> Stevo Slavic.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message