kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Cheng <wushuja...@gmail.com>
Subject Re: Lost message with Kafka configuration
Date Thu, 05 Jan 2017 08:53:12 GMT
kafka-console-producer.sh defaults to acks=0, which means that the producer essentially throws
messages at the broker and doesn't wait/retry to make sure they are properly received.

In the kafka-console-producer.sh usage text:
--request-required-acks <Integer:        The required acks of the producer
  request required acks>                   requests (default: 0)

Try re-running your test with "--request-required-acks -1" or  "--request-required-acks all"
(They are equivalent) This will tell the broker to wait for messages to be fully saved to
all replicas before returning an acknowledgement to the producer. You can read more about
acks in the producer configuration section of the kafka docs (http://kafka.apache.org/documentation/#producerconfigs
<http://kafka.apache.org/documentation/#producerconfigs>)

-James

> On Jan 4, 2017, at 1:25 AM, Hoang Bao Thien <hbthien0410@gmail.com> wrote:
> 
> Hi all,
> 
> I have a problem with losing messages from Kafka.
> The situation is as follows: I put a csv file with 286701 rows (size =
> 110MB)  into Kafka producer with command:
> $ cat test.csv | kafka-console-producer.sh --broker-list localhost:9092
> --topic MyTopic > /dev/null
> 
> and then count the number of lines from the Kafka consumer
> (kafka-console-consumer.sh --zookeeper localhost:2181 --topic MyTopic
> --from-beginning)
> However, I only get about 260K-270K, and this number of received messages
> changes for each test.
> 
> My configuration in the "config/server.properties" has some minor change
> compared to the original file:
> 
> log.retention.check.interval.hours=24
> log.retention.hours=168
> delete.topic.enable = true
> 
> The remaining configurations are the same as default value.
> 
> Could you please explain why the messages were lost in Kafka? And how to
> fix this problem please?
> 
> Thanks a lot.
> 
> Best regards
> ,
> Alex


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message