kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stevo Slavić <ssla...@gmail.com>
Subject Re: Blocked consumer - next offset same as current offset
Date Wed, 11 Nov 2015 11:15:23 GMT
Not sure if related, but this line in AbstractFetcherThread (Kafka 0.8.2
branch) looks suspicious
https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L122

This logic seems to have been changed significantly via
https://github.com/apache/kafka/commit/79f7cca85e9ed6511ad50cb9412bfa5c8e5b9ddb#diff-2d03a5d4349a23e56c09b4e773055249
for https://issues.apache.org/jira/browse/KAFKA-1461
so in Kafka 0.9 branch it looks like this
https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L132

Have dirty workaround for the moment, not trusting nextOffset info from
MessageAndOffset, transforming response to a new one with custom logic
calculating nextOffset (= offset used in fetch request + index of message
in message set). Will recheck at latest once 0.9 is fully released.

Unwanted behavior seems would get triggered only when consumer lag is of
appropriate size - I guess async producer made this issue more likely,
because of batch sending multiple messages to same partition.

On Wed, Nov 11, 2015 at 9:40 AM, Stevo Slavić <sslavic@gmail.com> wrote:

> Hello Apache Kafka community,
>
>
> I'm using simple consumer with Kafka 0.8.2.2 and noticed that under some
> conditions fetch response message set for a partition can contain at least
> one (if not all) MessageAndOffset with nextOffset being equal to current
> (committed) offset, offset used in fetch request. Not sure how it's related
> but I could notice this behavior especially often when I was using new
> async producer, and when fetch request was able to fetch several messages
> all the way to the end of the partition.
>
> Is this a feature or a bug?
>
>
> Kind regards,
>
> Stevo Slavic.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message