kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hoang Bao Thien <hbthien0...@gmail.com>
Subject Lost message with Kafka configuration
Date Wed, 04 Jan 2017 09:25:00 GMT
Hi all,

I have a problem with losing messages from Kafka.
The situation is as follows: I put a csv file with 286701 rows (size =
110MB)  into Kafka producer with command:
$ cat test.csv | kafka-console-producer.sh --broker-list localhost:9092
--topic MyTopic > /dev/null

and then count the number of lines from the Kafka consumer
(kafka-console-consumer.sh --zookeeper localhost:2181 --topic MyTopic
--from-beginning)
However, I only get about 260K-270K, and this number of received messages
changes for each test.

My configuration in the "config/server.properties" has some minor change
compared to the original file:

log.retention.check.interval.hours=24
log.retention.hours=168
delete.topic.enable = true

The remaining configurations are the same as default value.

Could you please explain why the messages were lost in Kafka? And how to
fix this problem please?

Thanks a lot.

Best regards
,
Alex

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message