kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Di Shang" <shan...@au1.ibm.com>
Subject Unable to recover from offline partition
Date Mon, 14 May 2018 04:13:08 GMT
Hi

We have recently upgraded from Kafka 0.10 to 1.1 , and we have encountered 
several occasions where some partitions in the cluster would go offline 
and unable to recover with the following error:

20:33:04.702 [controller-event-thread] ERROR state.change.logger - 
[Controller id=1 epoch=14] Controller 1 epoch 14 failed to change state 
for partition __consumer_offsets-39 from OfflinePartition to 
OnlinePartition
kafka.common.StateChangeFailedException: Failed to elect leader for 
partition __consumer_offsets-39 under strategy 
PreferredReplicaPartitionLeaderElectionStrategy
                 at 
kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:328)

~[kafka_2.11-1.1.0.jar:?]
                 at 
kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:326)

~[kafka_2.11-1.1.0.jar:?]
                 at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) 
~[scala-library-2.11.12.jar:?]
                 at 
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) 
~[scala-library-2.11.12.jar:?]
                 at 
kafka.controller.PartitionStateMachine.doElectLeaderForPartitions(PartitionStateMachine.scala:326)

~[kafka_2.11-1.1.0.jar:?]
                 at 
kafka.controller.PartitionStateMachine.electLeaderForPartitions(PartitionStateMachine.scala:254)

[kafka_2.11-1.1.0.jar:?]
                 at 
kafka.controller.PartitionStateMachine.doHandleStateChanges(PartitionStateMachine.scala:175)

[kafka_2.11-1.1.0.jar:?]
                 at 
kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:116)

[kafka_2.11-1.1.0.jar:?]
                 at 
kafka.controller.KafkaController.kafka$controller$KafkaController$$onPreferredReplicaElection(KafkaController.scala:604)

[kafka_2.11-1.1.0.jar:?]
                 at 
kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerAutoLeaderRebalance$3$$anonfun$apply$18.apply(KafkaController.scala:1000)

[kafka_2.11-1.1.0.jar:?]
                 at 
kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerAutoLeaderRebalance$3$$anonfun$apply$18.apply(KafkaController.scala:993)

[kafka_2.11-1.1.0.jar:?]
                 at 
scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:134) 
[scala-library-2.11.12.jar:?]
                 at 
scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:134) 
[scala-library-2.11.12.jar:?]
                 at 
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236) 
[scala-library-2.11.12.jar:?]
                 at 
scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) 
[scala-library-2.11.12.jar:?]
                 at 
scala.collection.mutable.HashMap$$anon$1.foreach(HashMap.scala:134) 
[scala-library-2.11.12.jar:?]
                 at 
kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerAutoLeaderRebalance$3.apply(KafkaController.scala:993)

[kafka_2.11-1.1.0.jar:?]
                 at 
kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerAutoLeaderRebalance$3.apply(KafkaController.scala:980)

[kafka_2.11-1.1.0.jar:?]
                 at 
scala.collection.immutable.Map$Map4.foreach(Map.scala:188) 
[scala-library-2.11.12.jar:?]
                 at 
kafka.controller.KafkaController.kafka$controller$KafkaController$$checkAndTriggerAutoLeaderRebalance(KafkaController.scala:980)

[kafka_2.11-1.1.0.jar:?]
                 at 
kafka.controller.KafkaController$AutoPreferredReplicaLeaderElection$.process(KafkaController.scala:1014)

[kafka_2.11-1.1.0.jar:?]
                 at 
kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply$mcV$sp(ControllerEventManager.scala:69)

[kafka_2.11-1.1.0.jar:?]
                 at 
kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:69)

[kafka_2.11-1.1.0.jar:?]
                 at 
kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:69)

[kafka_2.11-1.1.0.jar:?]
                 at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31) 
[kafka_2.11-1.1.0.jar:?]
                 at 
kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:68)

[kafka_2.11-1.1.0.jar:?]
                 at 
kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82) 
[kafka_2.11-1.1.0.jar:?]

We used to be able to fix offline partitions in 0.10 by restarting the 
whole cluster, and after the upgrade we have to revert 
unclean.leader.election.enable to true for the restart to work. 

My understanding is that doing unclean leader election could potentially 
lose data, the question is that is there an alternative way to fix offline 
partitions?



Thanks,

Di Shang




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message