kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun MA <mj.saber1...@gmail.com>
Subject Re: Halting because log truncation is not allowed for topic __consumer_offsets
Date Tue, 20 Dec 2016 12:23:31 GMT
Hi B,

Thanks for your reply. To clarify, in our case, it should be the leader which crash or something
goes wrong to the disk, not the followers (2 brokers that show’s the FATAL error)?


> On Dec 19, 2016, at 7:46 PM, Ben Stopford <ben@confluent.io> wrote:
> 
> Hi Jun
> 
> This should only be possible in situations where there is a crash or
> something happens to the underlying disks (assuming clean leader election).
> I've not come across others. The assumption, as I understand it, is that
> the underlying issue stems from KAFKA-1211
> <https://issues.apache.org/jira/browse/KAFKA-1211 <https://issues.apache.org/jira/browse/KAFKA-1211>>
which is being addressed
> in KIP-101
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-101+-+Alter+Replication+Protocol+to+use+Leader+Generation+rather+than+High+Watermark+for+Truncation
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-101+-+Alter+Replication+Protocol+to+use+Leader+Generation+rather+than+High+Watermark+for+Truncation>>.
> If you can reproduce in a more generally scenario we would be very
> interested.
> 
> All the best
> B
> 
> 
> On Mon, Dec 19, 2016 at 12:35 AM Jun MA <mj.saber1990@gmail.com <mailto:mj.saber1990@gmail.com>>
wrote:
> 
>> Would be grateful to hear opinions from experts out there. Thanks in
>> advance
>> 
>>> On Dec 17, 2016, at 11:06 AM, Jun MA <mj.saber1990@gmail.com> wrote:
>>> 
>>> Hi,
>>> 
>>> We saw the following FATAL error in 2 of our brokers (3 in total, the
>> active controller doesn’t have this) and they crashed in the same time.
>>> 
>>> [2016-12-16 16:12:47,085] FATAL [ReplicaFetcherThread-0-3], Halting
>> because log truncation is not allowed for topic __consumer_offsets, Current
>> leader 3's latest offset 5910081 is less than replica 1's latest offset
>> 5910082 (kafka.server.ReplicaFetcherThread)
>>> 
>>> Our solution is set topic __consumer_offsets
>> unclean.leader.election.enable=true temporarily, and restart brokers. In
>> this way we potentially lose offsets of some topics. Is there any better
>> solutions?
>>> 
>>> I saw related tickets https://issues.apache.org/jira/browse/KAFKA-3861 <
>> https://issues.apache.org/jira/browse/KAFKA-3861 <https://issues.apache.org/jira/browse/KAFKA-3861>>,
>> https://issues.apache.org/jira/browse/KAFKA-3410 <https://issues.apache.org/jira/browse/KAFKA-3410>
<
>> https://issues.apache.org/jira/browse/KAFKA-3410> and understand why
>> brokers crashed. But we didn’t see any scenarios mentioned in above
>> tickets. Is there any other reason why this happened? There’s no broker
>> restart involved in our case. How can we prevent it from happening?
>>> 
>>> We’re using 0.9.0 with unclean.leader.election.enable=false and
>> min.insync.replicas=2.
>>> 
>>> Thanks,
>>> Jun


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message