kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hoang Bao Thien <hbthien0...@gmail.com>
Subject Re: Lost message with Kafka configuration
Date Fri, 06 Jan 2017 13:02:33 GMT
It's perfect with the retries>0.

Thanks a lot, James.

Best regards

On Thu, Jan 5, 2017 at 10:51 PM, James Cheng <wushujames@gmail.com> wrote:

>
> > On Jan 5, 2017, at 8:23 AM, Hoang Bao Thien <hbthien0410@gmail.com>
> wrote:
> >
> > Yes, the problem is from producer configuration. And James Cheng has told
> > me how to fix it.
> > However I still get other poblem with a large file:
> >
> > org.apache.kafka.common.errors.TimeoutException: Batch containing 36
> > record(s) expired due to timeout while requesting metadata from brokers
> for
> > MyTopic-0
> >
>
> kafka-console-producer.sh defaults to retries=0. If there is a timeout, as
> that error indicates, I think it drops the messages it was trying to send.
>
> As a test, try setting retries to something high, by doing
> "--producer-property retries=<somebignumber>"
>
> See the description of "retries" at http://kafka.apache.org/
> documentation/#producerconfigs <http://kafka.apache.org/
> documentation/#producerconfigs>.
>
> -James
>
>
> > Best regards,
> >
> > On Thu, Jan 5, 2017 at 10:23 AM, Protoss Hu <hbprotoss@yahoo.com.invalid
> >
> > wrote:
> >
> >> You mean the messages were lost on the way to broker before the broker
> >> actually received?
> >>
> >> Protoss Hu
> >> Blog: http://hbprotoss.github.io/
> >> Weibo: http://weibo.com/hbprotoss
> >>
> >> 2017年1月5日 +0800 PM4:53 James Cheng <wushujames@gmail.com>,写道:
> >>> kafka-console-producer.sh defaults to acks=0, which means that the
> >> producer essentially throws messages at the broker and doesn't
> wait/retry
> >> to make sure they are properly received.
> >>>
> >>> In the kafka-console-producer.sh usage text:
> >>> --request-required-acks <Integer: The required acks of the producer
> >>> request required acks> requests (default: 0)
> >>>
> >>> Try re-running your test with "--request-required-acks -1" or
> >> "--request-required-acks all" (They are equivalent) This will tell the
> >> broker to wait for messages to be fully saved to all replicas before
> >> returning an acknowledgement to the producer. You can read more about
> acks
> >> in the producer configuration section of the kafka docs (
> >> http://kafka.apache.org/documentation/#producerconfigs <
> >> http://kafka.apache.org/documentation/#producerconfigs>)
> >>>
> >>> -James
> >>>
> >>>> On Jan 4, 2017, at 1:25 AM, Hoang Bao Thien <hbthien0410@gmail.com>
> >> wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>> I have a problem with losing messages from Kafka.
> >>>> The situation is as follows: I put a csv file with 286701 rows (size
=
> >>>> 110MB) into Kafka producer with command:
> >>>> $ cat test.csv | kafka-console-producer.sh --broker-list
> localhost:9092
> >>>> --topic MyTopic > /dev/null
> >>>>
> >>>> and then count the number of lines from the Kafka consumer
> >>>> (kafka-console-consumer.sh --zookeeper localhost:2181 --topic MyTopic
> >>>> --from-beginning)
> >>>> However, I only get about 260K-270K, and this number of received
> >> messages
> >>>> changes for each test.
> >>>>
> >>>> My configuration in the "config/server.properties" has some minor
> >> change
> >>>> compared to the original file:
> >>>>
> >>>> log.retention.check.interval.hours=24
> >>>> log.retention.hours=168
> >>>> delete.topic.enable = true
> >>>>
> >>>> The remaining configurations are the same as default value.
> >>>>
> >>>> Could you please explain why the messages were lost in Kafka? And how
> >> to
> >>>> fix this problem please?
> >>>>
> >>>> Thanks a lot.
> >>>>
> >>>> Best regards
> >>>> ,
> >>>> Alex
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message