kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Richardson <ch...@chrisrichardson.net>
Subject Kafka cluster behavior: broker no longer belonging to ISR, offsets reset to middle of partition
Date Mon, 19 Sep 2016 21:50:04 GMT

One of my Kafka clusters (3 brokers,
default.replication.factor=2) that has been working fine until yesterday.
The message volume was pretty low. There were no obvious problems except....

The first symptom was *kafka-consumer-groups.sh* failing with an empty.head

When I used *kafka-topics --describe* I saw that one of the brokers was no
longer part of the appropriate ISRs.
Restarting that broker appeared not solve the problem.
In fact, I got the impression that the broker was temporarily in the ISR
and then left again.

I think I restarted each broker and eventually things returned to normal.
The problem then reoccurred a couple of hours later.

During this time, I also had a problem with one of the Kafka

ERROR kafka.consumer.ConsumerFetcherThread -
Current offset 5488 for partition [mytopic,5] out of range; reset offset to

This topic partition had >5488 messages so there offset was definitely not
out of range. The result was that the consumer reprocessed old messages.
The lag as reported by kafka-consumer-groups.sh when from <10 to > 2500

Thoughts? Recommendations for debugging this problem when it occurs again?


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message