kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandeep Bishnoi <sandeepbishnoi.b...@gmail.com>
Subject Failure in Leader Election on broker shutdown
Date Fri, 19 Jun 2015 21:37:31 GMT
Hi,

 I have a kafka cluster of three nodes.

 I have constructed a topic with the following command:
 bin/kafka-topics.sh --create --zookeeper localhost:2181
--replication-factor 1 --partitions 3 --topic testv1p3

 So the topic "testv1p3" has 3 partitions and replication factor is 1.
 Here is the result of describe command:
  kafka_2.10-0.8.2.0]$ bin/kafka-topics.sh --describe --zookeeper
localhost:2181 --topic testv1p3

Topic:testv1p3    PartitionCount:3    ReplicationFactor:1    Configs:
    Topic: testv1p3    Partition: 0    Leader: 1    Replicas: 1    Isr: 1
    Topic: testv1p3    Partition: 1    Leader: 2    Replicas: 2    Isr: 2
    Topic: testv1p3    Partition: 2    Leader: 0    Replicas: 0    Isr: 0

So far things are good.

Now I tried to kill a broker using bin/kafka-server-stop.sh
The broker was stopped successfully.

 Now I wanted to ensure that there is a new leader for the partition which
was hosted on the terminated broker.
 Here is the output of describe command post broker termination:
 Topic:testv1p3    PartitionCount:3    ReplicationFactor:1    Configs:
    Topic: testv1p3    Partition: 0    Leader: 1    Replicas: 1    Isr: 1
    Topic: testv1p3    Partition: 1    Leader: -1    Replicas: 2    Isr:
    Topic: testv1p3    Partition: 2    Leader: 0    Replicas: 0    Isr: 0

Leader for partition:1 is -1.

Java API for kafka returns null for leader() in PartitionMetadata for
partition 1.

When I restarted the broker which was stopped earlier.
Things go back to normal.

1) Does leader selection happen automatically ?
2) If yes, do I need any particular configuration in broker or topic config
?
3) If not, what is command to ensure that I have a leader for partition 1
in case its lead broker goes down.
 FYI I tried to run bin/bin/kafka-preferred-replica-election.sh --zookeeper
localhost:2181
 Post this script run, the topic description still remains same and no
leader for partition 1.

It will be great to get any help on this.


Reference:
Console log for (kafka-server-stop.sh):
[2015-06-19 14:25:00,241] INFO [Kafka Server 2], shutting down
(kafka.server.KafkaServer)
[2015-06-19 14:25:00,243] INFO [Kafka Server 2], Starting controlled
shutdown (kafka.server.KafkaServer)
[2015-06-19 14:25:00,267] INFO [Kafka Server 2], Controlled shutdown
succeeded (kafka.server.KafkaServer)
[2015-06-19 14:25:00,273] INFO Deregistered broker 2 at path
/brokers/ids/2. (kafka.utils.ZkUtils$)
[2015-06-19 14:25:00,274] INFO [Socket Server on Broker 2], Shutting down
(kafka.network.SocketServer)
[2015-06-19 14:25:00,279] INFO [Socket Server on Broker 2], Shutdown
completed (kafka.network.SocketServer)
[2015-06-19 14:25:00,280] INFO [Kafka Request Handler on Broker 2],
shutting down (kafka.server.KafkaRequestHandlerPool)
[2015-06-19 14:25:00,282] INFO [Kafka Request Handler on Broker 2], shut
down completely (kafka.server.KafkaRequestHandlerPool)
[2015-06-19 14:25:00,600] INFO [Replica Manager on Broker 2]: Shut down
(kafka.server.ReplicaManager)
[2015-06-19 14:25:00,601] INFO [ReplicaFetcherManager on broker 2] shutting
down (kafka.server.ReplicaFetcherManager)
[2015-06-19 14:25:00,602] INFO [ReplicaFetcherManager on broker 2] shutdown
completed (kafka.server.ReplicaFetcherManager)
[2015-06-19 14:25:00,604] INFO [Replica Manager on Broker 2]: Shut down
completely (kafka.server.ReplicaManager)
[2015-06-19 14:25:00,605] INFO Shutting down. (kafka.log.LogManager)
[2015-06-19 14:25:00,618] INFO Shutdown complete. (kafka.log.LogManager)
[2015-06-19 14:25:00,620] WARN Kafka scheduler has not been started
(kafka.utils.Utils$)
java.lang.IllegalStateException: Kafka scheduler has not been started
    at kafka.utils.KafkaScheduler.ensureStarted(KafkaScheduler.scala:114)
    at kafka.utils.KafkaScheduler.shutdown(KafkaScheduler.scala:86)
    at
kafka.controller.KafkaController.onControllerResignation(KafkaController.scala:350)
    at kafka.controller.KafkaController.shutdown(KafkaController.scala:664)
    at
kafka.server.KafkaServer$$anonfun$shutdown$9.apply$mcV$sp(KafkaServer.scala:287)
    at kafka.utils.Utils$.swallow(Utils.scala:172)
    at kafka.utils.Logging$class.swallowWarn(Logging.scala:92)
    at kafka.utils.Utils$.swallowWarn(Utils.scala:45)
    at kafka.utils.Logging$class.swallow(Logging.scala:94)
    at kafka.utils.Utils$.swallow(Utils.scala:45)
    at kafka.server.KafkaServer.shutdown(KafkaServer.scala:287)
    at
kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:42)
    at kafka.Kafka$$anon$1.run(Kafka.scala:42)
[2015-06-19 14:25:00,623] INFO Terminate ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
[2015-06-19 14:25:00,625] INFO Session: 0x14de8e5f2b801f7 closed
(org.apache.zookeeper.ZooKeeper)
[2015-06-19 14:25:00,625] INFO EventThread shut down
(org.apache.zookeeper.ClientCnxn)
[2015-06-19 14:25:00,625] INFO [Kafka Server 2], shut down completed
(kafka.server.KafkaServer)


Regards,
Sandeep

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message