kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Aley <Thomas.A...@ibm.com>
Subject Re: When a broker down, Producer LOST messages!
Date Thu, 01 Mar 2018 09:21:27 GMT

The log is not attached. I'm assuming your topic has a replication factor 
greater than 1 so that it is available from another Broker if the 
partition leader fails. Try adding 

props.put("acks", "all");

to your producer and run your experiment again. If you configured your 
topic to have --replication-factor 3 and your brokers (or topic itself) is 
configured with min.insync.replicas=2 for example then your producer will 
require acknowledgment of receipt of each message from 2 of your 3 brokers 
with the 'acks=all' property in use making your topic resilient. 

Hope this helps,

Tom Aley

From:   "许志峰" <zhifengxucs@gmail.com>
To:     users@kafka.apache.org
Date:   01/03/2018 08:59
Subject:        When a broker down, Producer LOST messages!

Hi all,

I have a kafka cluster with 3 nodes: node1, node2, node3

kafka version is, which I can not change!

A Producer writes msg to kafka, its code framework is like this in 
Properties props = new Properties();
props.put("bootstrap.servers", "node1:9092,node2:9092,node3:9092");
props.put(key serializer and value serializer);

for(i = 0; i < 10; ++i){
    producer = new Producer(props);
    msg = "this is msg " + i;

After the first 4 messages are send successfully, I killed broker on 
node1.  the 5th and 6th messages are LOST.

Producer first get the broker list from the PropertyConfig, i.e. [node1, 
node2, node3], then producer choose one broker, connect with it and get 
METADATA from it. 

I heard that when one broker in the list is unavailable, the kafka client 
will change to another 

But in my case, if the broker choose node1, which is already dead, it will 
get a Fetch MetaData Timeout Exception and STOPPED! msg is not writed into 

Attached is the complete Log. you can only focus on the colorful lines.  

you can see that, I wrote 10 msgs to Kafka, the first 4 succeed, when I 
kill one broker, msg5 and msg6 are LOST, because the choose NODE1, 
msg7,8,9,10 are succeed because they did not choose node1.

I checkout the Kafka source codes and get nothing.

Do anybody know the reason?  where are the related classes/functions 
located in the source code?

Any clue will be appreciated!

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message