Hello All, I'm using "librdkafka" for my C Project which has is needed to support geo redundant kafka process (zookeeper + Broker). Machine 1 : Producer1 : Broker IP"sysctrl1.vsepx.broker.com:9092, sysctrl2.vsepx.broker.com:9092" Machine 2 : Broker1 : advertised.host.name=sysctrl1.vsepx.broker.com Machine 3 : Broker2 : advertised.host.name=sysctrl2.vsepx.broker.com Scenario: 1. Broker1 is UP, Broker 2 is Down. 2. Producer1 is sending messages to Broker1 successfully 3. Broker1 goes down. 4. Broker2 comes UP 5. Producer1 connection with broker 1 fails, But Connection with Broker2 is established. 6. Producer1 exchanges meta data with Broker2, But messages are not being sent to Broker2, Instead the producer1 is still trying to send the same to Broker1 (which is down). 7. All Messages Fail. Now, If I kill and restart Producer1, then all the new messages are succesfully sent to Broker2. Broker logs: [2015-12-03 09:28:24,573] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) [2015-12-03 09:28:24,579] INFO Registered broker 0 at path /brokers/ids/0 with address sysctrl2.vsepx.broker.com:9092. (kafka.utils.ZkUtils$) [2015-12-03 09:28:24,593] INFO [Kafka Server 0], started (kafka.server.KafkaServer) [2015-12-03 09:28:24,718] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [topic1,1],[topic1,0],[topic1,10],[topic1,15],[topic1,9],[topic1,4],[topic1,8],[topic1,3],[topic1,13],[topic1,12],[topic1,2],[topic1,6],[topic1,7],[topic1,5],[topic1,11], topic1,14 [2015-12-03 09:28:24,982] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [topic1,1],[topic1,0],[topic1,10],[topic1,15],[topic1,9],[topic1,4],[topic1,8],[topic1,3],[topic1,13],[topic1,12],[topic1,2],[topic1,6],[topic1,7],[topic1,5],[topic1,11], topic1,14 [2015-12-03 09:30:10,195] INFO Closing socket connection to /12.1.1.81. (kafka.network.Processor) Below are few logs from sample stub code % Sent 3 bytes to topic topic1 partition -1, outQlen[127] log_cb : rdkafka#producer-0: [7] [RECV] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Received MetadataResponse (473 bytes, CorrId 12, rtt 142.35ms)] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: ===== Received metadata from sysctrl2.vsepx.broker.com:9092/bootstrap =====] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: 1 brokers, 1 topics] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Broker #0/1: sysctrl2.vsepx.broker.com:9092 NodeId 0] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Topic #0/1: topic1 with 16 partitions] log_cb : rdkafka#producer-0: [7] [PARTCNT] [No change in partition count for topic topic1] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 8 Leader 0] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [8] with 8 messages (19 bytes) queued] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 11 Leader 0] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [11] with 8 messages (19 bytes) queued] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 2 Leader 0] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [2] with 8 messages (19 bytes) queued] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 5 Leader 0] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [5] with 8 messages (20 bytes) queued] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 14 Leader 0] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [14] with 8 messages (19 bytes) queued] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 4 Leader 0] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [4] with 8 messages (20 bytes) queued] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 13 Leader 0] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [13] with 8 messages (19 bytes) queued] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 7 Leader 0] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [7] with 7 messages (17 bytes) queued] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 1 Leader 0] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [1] with 8 messages (19 bytes) queued] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 10 Leader 0] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [10] with 8 messages (19 bytes) queued] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 9 Leader 0] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [9] with 8 messages (19 bytes) queued] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 3 Leader 0] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [3] with 8 messages (19 bytes) queued] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 12 Leader 0] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [12] with 8 messages (19 bytes) queued] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 15 Leader 0] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [15] with 8 messages (19 bytes) queued] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 6 Leader 0] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [6] with 8 messages (20 bytes) queued] log_cb : rdkafka#producer-0: [7] [METADATA] [ sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 0 Leader 0] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [0] with 8 messages (19 bytes) queued] log_cb : rdkafka#producer-0: [7] [PARTCNT] [Partitioning 0 unassigned messages in topic topic1 to 16 partitions] log_cb : rdkafka#producer-0: [7] [UAS] [0/0 messages were partitioned in topic topic1] log_cb : rdkafka#producer-0: [7] [CONNECT] [sysctrl1.vsepx.broker.com:9092/0: broker in state DOWN connecting] log_cb : rdkafka#producer-0: [7] [CONNECT] [ sysctrl1.vsepx.broker.com:9092/bootstrap: broker in state DOWN connecting] log_cb : rdkafka#producer-0: [7] [CONNECT] [ sysctrl1.vsepx.broker.com:9092/bootstrap: couldn't connect to ipv4# 192.168.108.140:9092: Connection refused] log_cb : rdkafka#producer-0: [7] [BROKERFAIL] [ sysctrl1.vsepx.broker.com:9092/bootstrap: failed: err: Local: Communication failure with broker: (errno: Connection refused)] log_cb : rdkafka#producer-0: [7] [BUFQ] [ sysctrl1.vsepx.broker.com:9092/bootstrap: Purging bufq with 0 buffers] log_cb : rdkafka#producer-0: [7] [CONNECT] [sysctrl1.vsepx.broker.com:9092/0: couldn't connect to ipv4#192.168.108.140:9092: Connection refused] log_cb : rdkafka#producer-0: [7] [BROKERFAIL] [ sysctrl1.vsepx.broker.com:9092/0: failed: err: Local: Communication failure with broker: (errno: Connection refused)] log_cb : rdkafka#producer-0: [7] [BUFQ] [sysctrl1.vsepx.broker.com:9092/0: Purging bufq with 0 buffers] log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [8]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [8]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [8]] log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [11]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [11]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [11]] log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [2]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [2]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [2]] log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [5]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [5]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [5]] log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [14]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [14]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [14]] log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [4]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [4]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [4]] log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [13]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [13]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [13]] log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [7]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [7]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [7]] log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [1]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [1]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [1]] log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [10]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [10]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [10]] log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [9]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [9]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [9]] log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [3]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [3]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [3]] log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [12]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [12]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [12]] log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [15]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [15]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [15]] log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [6]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [6]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [6]] log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [0]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [0]] log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [0]] Regards, Mazhar