kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kris K <squareksc...@gmail.com>
Subject duplicate messages at consumer
Date Wed, 17 Jun 2015 04:06:09 GMT

While testing message delivery using kafka, I realized that few duplicate
messages got delivered by the consumers in the same consumer group (two
consumers got the same message with few milli-seconds difference). However,
I do not see any redundancy at the producer or broker. One more observation
is that - this is not happening when I use only one consumer thread.

I am running 3 brokers ( with 3 Zookeeper nodes. There are 3
partitions in the topic and replication-factor is 3. For producing, am
using New Producer with compression.type=none.

On the consumer end, I have 3 High level consumers in the same consumer
group running with one consumer thread each, on three different hosts. Auto
commit is set to true for consumer.

Size of each message would range anywhere between 0.7 KB and  2 MB. The max
volume for this test is 100 messages/hr.

I looked at controller log for any possibility of consumer rebalance during
this time, but did not find any. In the server log of all the brokers the
error - java.io.IOException: Connection reset by peer is almost being
written continuously.

So, is it possible to achieve exactly-once delivery with the current high
level consumer without needing an extra layer to remove redundancy?

Could you please point me to any settings or logs that would help me tune
the configuration ?

*PS: I tried searching for similar discussions, but could not find any. If
its already been answered, please provide the link.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message