kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Shannon <adam.shan...@banno.com>
Subject Re: duplicate messages at consumer
Date Wed, 17 Jun 2015 15:24:49 GMT
This is actually an expected consequence of using distributed systems. The
kafka FAQ has a good answer


On Tue, Jun 16, 2015 at 11:06 PM, Kris K <squarekscode@gmail.com> wrote:

> Hi,
> While testing message delivery using kafka, I realized that few duplicate
> messages got delivered by the consumers in the same consumer group (two
> consumers got the same message with few milli-seconds difference). However,
> I do not see any redundancy at the producer or broker. One more observation
> is that - this is not happening when I use only one consumer thread.
> I am running 3 brokers ( with 3 Zookeeper nodes. There are 3
> partitions in the topic and replication-factor is 3. For producing, am
> using New Producer with compression.type=none.
> On the consumer end, I have 3 High level consumers in the same consumer
> group running with one consumer thread each, on three different hosts. Auto
> commit is set to true for consumer.
> Size of each message would range anywhere between 0.7 KB and  2 MB. The max
> volume for this test is 100 messages/hr.
> I looked at controller log for any possibility of consumer rebalance during
> this time, but did not find any. In the server log of all the brokers the
> error - java.io.IOException: Connection reset by peer is almost being
> written continuously.
> So, is it possible to achieve exactly-once delivery with the current high
> level consumer without needing an extra layer to remove redundancy?
> Could you please point me to any settings or logs that would help me tune
> the configuration ?
> *PS: I tried searching for similar discussions, but could not find any. If
> its already been answered, please provide the link.
> Thanks,
> Kris

Adam Shannon | Software Engineer | Banno | Jack Henry
206 6th Ave Suite 1020 | Des Moines, IA 50309 | Cell: 515.867.8337

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message