kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Wu <stevenz...@gmail.com>
Subject some producers stuck when one broker is bad
Date Wed, 09 Sep 2015 03:26:38 GMT
We have observed that some producer instances stopped sending traffic to
brokers, because the memory buffer is full. those producers got stuck in
this state permanently. Because we couldn't find out which broker is bad
here. So I did a rolling restart the all brokers. after the bad broker got
bounce, those stuck producers out of the woods automatically.

I don't know the exact problem with that bad broker. it seems to me that
some ZK states are inconsistent.

I know timeout fix from KAFKA-2120 can probably avoid the permanent stuck.
Here are some additional questions.
1) any suggestion on how to identify the bad broker(s)?
2) why bouncing of the bad broker got the producers recovered automatically
(without restarting producers)

producer: 0.8.2.1
broker: 0.8.2.1

Thanks,
Steven

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message