kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahmed H." <ahmed.ham...@gmail.com>
Subject Kafka rebalancing causes Zookeeper to fail
Date Wed, 22 Jan 2014 14:24:17 GMT
I have a basic Zookeeper/Kafka setup. I am still on Kafka 0.8 beta 1, and
Zookeeper 3.4.5. The activity on this machine isn't massive...I would say
the Kafka queues get a consistent 1 message every 2-3 seconds, as well as
occasional spikes, but still nothing large enough to push the limits. Both
Kafka and Zookeeper are running on the same machine.

Occasionally, a rebalance is triggered, which causes our Kafka clients to
try reconnecting several times, but it ultimately fails with the following
error:


04:56:10,020 INFO  [kafka.consumer.ZookeeperConsumerConnector]
(alarms.topology.updates_<host>-1383643783747-c7775701_watcher_executor)
[alarms.topology.updates_<host>-1383643783747-c7775701], exception
during rebalance : org.I0Itec.zkclient.exception.ZkNoNodeException:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode
= NoNode for /consumers/alarms.topology.updates/ids/alarms.topology.updates_<host>-1383643783747-c7775701
	at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
[zkclient-0.3.jar:0.3]
	at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
[zkclient-0.3.jar:0.3]
	at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
[zkclient-0.3.jar:0.3]
	at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
[zkclient-0.3.jar:0.3]
	at kafka.utils.ZkUtils$.readData(ZkUtils.scala:407)
[kafka_2.9.2-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
	at kafka.consumer.TopicCount$.constructTopicCount(TopicCount.scala:52)
[kafka_2.9.2-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
	at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(ZookeeperConsumerConnector.scala:401)
[kafka_2.9.2-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
	at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:374)
[kafka_2.9.2-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
[scala-library-2.9.2.jar:]
	at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:369)
[kafka_2.9.2-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
	at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:326)
[kafka_2.9.2-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/consumers/alarms.topology.updates/ids/alarms.topology.updates_<host>-1383643783747-c7775701
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
[zookeeper-3.4.3.jar:3.4.3-1240972]
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[zookeeper-3.4.3.jar:3.4.3-1240972]
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1131)
[zookeeper-3.4.3.jar:3.4.3-1240972]
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1160)
[zookeeper-3.4.3.jar:3.4.3-1240972]
	at org.I0Itec.zkclient.ZkConnection.readData(ZkConnection.java:103)
[zkclient-0.3.jar:0.3]
	at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:770)
[zkclient-0.3.jar:0.3]
	at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:766)
[zkclient-0.3.jar:0.3]
	at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
[zkclient-0.3.jar:0.3]
	... 9 more


Our Kafka consumers are written in Clojure (
https://github.com/pingles/clj-kafka).

Any ideas on what can cause such behaviour? The rebalances themselves
happen sporadically, but when they do, they sometimes fail and an error
like the one above is shown. I'm not sure if this is a Kafka or Zookeeper
problem at this point, but any help would be appreciated.

Thanks

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message