kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Protoss Hu <hbprot...@yahoo.com.INVALID>
Subject Re: Lost message with Kafka configuration
Date Thu, 05 Jan 2017 09:23:54 GMT
You mean the messages were lost on the way to broker before the broker actually received?

Protoss Hu
Blog: http://hbprotoss.github.io/
Weibo: http://weibo.com/hbprotoss

2017年1月5日 +0800 PM4:53 James Cheng <wushujames@gmail.com>,写道:
> kafka-console-producer.sh defaults to acks=0, which means that the producer essentially
throws messages at the broker and doesn't wait/retry to make sure they are properly received.
>
> In the kafka-console-producer.sh usage text:
> --request-required-acks <Integer: The required acks of the producer
> request required acks> requests (default: 0)
>
> Try re-running your test with "--request-required-acks -1" or "--request-required-acks
all" (They are equivalent) This will tell the broker to wait for messages to be fully saved
to all replicas before returning an acknowledgement to the producer. You can read more about
acks in the producer configuration section of the kafka docs (http://kafka.apache.org/documentation/#producerconfigs
<http://kafka.apache.org/documentation/#producerconfigs>)
>
> -James
>
> > On Jan 4, 2017, at 1:25 AM, Hoang Bao Thien <hbthien0410@gmail.com> wrote:
> >
> > Hi all,
> >
> > I have a problem with losing messages from Kafka.
> > The situation is as follows: I put a csv file with 286701 rows (size =
> > 110MB) into Kafka producer with command:
> > $ cat test.csv | kafka-console-producer.sh --broker-list localhost:9092
> > --topic MyTopic > /dev/null
> >
> > and then count the number of lines from the Kafka consumer
> > (kafka-console-consumer.sh --zookeeper localhost:2181 --topic MyTopic
> > --from-beginning)
> > However, I only get about 260K-270K, and this number of received messages
> > changes for each test.
> >
> > My configuration in the "config/server.properties" has some minor change
> > compared to the original file:
> >
> > log.retention.check.interval.hours=24
> > log.retention.hours=168
> > delete.topic.enable = true
> >
> > The remaining configurations are the same as default value.
> >
> > Could you please explain why the messages were lost in Kafka? And how to
> > fix this problem please?
> >
> > Thanks a lot.
> >
> > Best regards
> > ,
> > Alex
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message