kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun MA <mj.saber1...@gmail.com>
Subject Halting because log truncation is not allowed for topic __consumer_offsets
Date Sat, 17 Dec 2016 03:06:21 GMT

We saw the following FATAL error in 2 of our brokers (3 in total, the active controller doesn’t
have this) and they crashed in the same time. 

[2016-12-16 16:12:47,085] FATAL [ReplicaFetcherThread-0-3], Halting because log truncation
is not allowed for topic __consumer_offsets, Current leader 3's latest offset 5910081 is less
than replica 1's latest offset 5910082 (kafka.server.ReplicaFetcherThread)

Our solution is set topic __consumer_offsets unclean.leader.election.enable=true temporarily,
and restart brokers. In this way we potentially lose offsets of some topics. Is there any
better solutions?

I saw related tickets https://issues.apache.org/jira/browse/KAFKA-3861 <https://issues.apache.org/jira/browse/KAFKA-3861>,
https://issues.apache.org/jira/browse/KAFKA-3410 <https://issues.apache.org/jira/browse/KAFKA-3410>
and understand why brokers crashed. But we didn’t see any scenarios mentioned in above tickets.
Is there any other reason why this happened? There’s no broker restart involved in our case.
How can we prevent it from happening?

We’re using 0.9.0 with unclean.leader.election.enable=false and min.insync.replicas=2.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message