kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brett Rann <br...@zendesk.com.INVALID>
Subject Re: Insanely long recovery time with Kafka
Date Fri, 05 Jan 2018 23:02:37 GMT
What do the broker logs say its doing during all that time?

There are some consumer offset / log cleaner bugs which caused us similarly
log delays. that was easily visible by watching the log cleaner activity in
the logs, and in our monitoring of partition sizes watching them go down,
along with IO activity on the host for those files.

On Sat, Jan 6, 2018 at 7:48 AM, Vincent Rischmann <vincent@rischmann.fr>

> Hello,
> so I'm upgrading my brokers from to to fix this bug
> https://issues.apache.org/jira/browse/KAFKA-4523
> <https://issues.apache.org/jira/browse/KAFKA-4523>
> Unfortunately while stopping one broker, it crashed exactly because of
> this bug. No big deal usually, except after restarting Kafka in
> the recovery is taking a really long time.
> I have around 6TB of data on that broker, and before when it crashed it
> usually took around 30 to 45 minutes to recover, but now I'm at almost
> 5h since Kafka started and it's still not recovered.
> I'm wondering what could have changed to have such a dramatic effect on
> recovery time ? Is there maybe something I can tweak to try to reduce
> the time ?
> Thanks.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message