kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 许志峰 <zhifengx...@gmail.com>
Subject Re: When a broker down, Producer LOST messages!
Date Thu, 01 Mar 2018 09:35:04 GMT
Hi,

Thank you for you reply.

It looks like  the attachment is fine. I have a replication-factor of 3.
The producer can not even get the METADATA from a broker(when it chooses a
dead broker), let alone sending messages and wait for all brokers to ACK.

Best regards!

2018-03-01 17:21 GMT+08:00 Thomas Aley <Thomas.Aley@ibm.com>:

> Hi,
>
> The log is not attached. I'm assuming your topic has a replication factor
> greater than 1 so that it is available from another Broker if the
> partition leader fails. Try adding
>
> props.put("acks", "all");
>
> to your producer and run your experiment again. If you configured your
> topic to have --replication-factor 3 and your brokers (or topic itself) is
> configured with min.insync.replicas=2 for example then your producer will
> require acknowledgment of receipt of each message from 2 of your 3 brokers
> with the 'acks=all' property in use making your topic resilient.
>
> Hope this helps,
>
> Tom Aley
> thomas.aley@ibm.com
>
>
>
> From:   "许志峰" <zhifengxucs@gmail.com>
> To:     users@kafka.apache.org
> Date:   01/03/2018 08:59
> Subject:        When a broker down, Producer LOST messages!
>
>
>
> Hi all,
>
>
> I have a kafka cluster with 3 nodes: node1, node2, node3
>
> kafka version is 0.8.2.1, which I can not change!
>
> A Producer writes msg to kafka, its code framework is like this in
> pseudo-code:
> Properties props = new Properties();
> props.put("bootstrap.servers", "node1:9092,node2:9092,node3:9092");
> props.put(key serializer and value serializer);
>
> for(i = 0; i < 10; ++i){
>     producer = new Producer(props);
>     msg = "this is msg " + i;
>     producer.send(msg);
>     producer.close()
> }
>
> After the first 4 messages are send successfully, I killed broker on
> node1.  the 5th and 6th messages are LOST.
>
> Producer first get the broker list from the PropertyConfig, i.e. [node1,
> node2, node3], then producer choose one broker, connect with it and get
> METADATA from it.
>
> I heard that when one broker in the list is unavailable, the kafka client
> will change to another
>
> But in my case, if the broker choose node1, which is already dead, it will
> get a Fetch MetaData Timeout Exception and STOPPED! msg is not writed into
> Kafka.
>
>
> Attached is the complete Log. you can only focus on the colorful lines.
>
> you can see that, I wrote 10 msgs to Kafka, the first 4 succeed, when I
> kill one broker, msg5 and msg6 are LOST, because the choose NODE1,
> msg7,8,9,10 are succeed because they did not choose node1.
>
> I checkout the Kafka source codes and get nothing.
>
> Do anybody know the reason?  where are the related classes/functions
> located in the source code?
>
> Any clue will be appreciated!
>
>
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message