kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koushik Chitta <kchi...@microsoft.com.INVALID>
Subject Curious case of under min ISR before offline partition
Date Wed, 28 Aug 2019 18:02:46 GMT
Hi All,

We had a topic partition(with 5 replication) going offline when leader of the partition was
down. Below is some analysis

Kafka server - 1.1  , relevant config  (replica.fetch.wait.max.ms = 500, replica.fetch.min.bytes
= 50000, replica.lag.time.max.ms=10000)
Topic partition (Test.Request-3) - replication 5 ,  Replica List [17, 425222741, 425222681,
423809494,425222740]  , unclean leader election = false

Sequence of events.


  1.  Leader(425222740) of the partition is down.



  1.  Controller detects the offline broker.

[2019-08-26 13:00:22,037] INFO [Controller id=423809469] Newly added brokers: , deleted brokers:
425222740, all live brokers: 15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,423809443,423809444,423809450,423809458,423809463,423809464,423809469,423809474,423809494,425218574,425222675,425222681,425222741,425222745
(kafka.controller.KafkaController)



  1.  Controller sends update metadata request , but observes only leader in the isr. Please
note that none of the isr metrics "underminisr", "isrshirnk" are captured before or after
the partition going offline.
[2019-08-26 13:00:05,333] TRACE [Controller id=423809469 epoch=206] Sending UpdateMetadata
request PartitionState(controllerEpoch=204, leader=425222740, leaderEpoch=435, isr=[425222740],
zkVersion=804, replicas=[425222740, 423809494, 17, 425222741, 425222681], offlineReplicas=[])
to brokers Set(425222740, 423809458, 425222741, 24, 425218574, 25, 423809474, 26, 27, 19,
20, 21, 22, 425222681, 23, 423809450, 15, 16, 423809443, 423809494, 425222675, 17, 423809444,
18, 423809469, 425222745, 423809463, 28, 423809464, 29) for Test.Request-3 (state.change.logger)


  1.  New leader election fails under strategy OfflinePartitionLeaderElectionStrategy since
no replicas are in ISR list.



  1.  All replicas see replica fetch request errors as they cannot connect to leader.

Any pointers on why the ISR list was shrunk just before the leader went down forcing the partition
to go offline.

Thanks,
Koushik

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message