kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Santilli <jonathansanti...@gmail.com>
Subject Re: Doubts in Kafka
Date Tue, 16 Jul 2019 08:50:09 GMT
Hello Aruna,

if the duplication you are referring to is the duplication of the
events/records that arrive and are consumed to/from Kafka, exactly-once
semantics and transactions are what you are looking for.
Kafka is prepared (since version 0.11 IIRC) to support exactly once, it
means that events are going to be sent by the Producer to the Broker just
once, and will be consumed/processed by the Consumer just when they are
ready to be consumed, once.
In order to guaranty that the records are going to be produced and consumed
just once, both, Producer and Consumer need to be prepared/configured
correctly.

Take a look at the documentation about that, also, this posts explains
nicely how it works:
https://www.confluent.io/blog/transactions-apache-kafka/
https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/

Hope that helps, cheers!
--
Jonathan




On Tue, Jul 16, 2019 at 5:46 AM aruna ramachandran <arunaeienec@gmail.com>
wrote:

> I want to process a single message at a time to avoid duplication.
>
>
> On Mon, Jul 15, 2019 at 9:45 PM Pere Urbón Bayes <pere.urbon@gmail.com>
> wrote:
>
> > The good question, Aruna, is why would you like to do that?
> >
> > -- Pere
> >
> > Missatge de aruna ramachandran <arunaeienec@gmail.com> del dia dl., 15
> de
> > jul. 2019 a les 14:17:
> >
> > > Is there a way to get the Consumer to only read one message at a time
> and
> > > commit the offset after processing a single message.
> > >
> >
> >
> > --
> > Pere Urbon-Bayes
> > Software Architect
> > http://www.purbon.com
> > https://twitter.com/purbon
> > https://www.linkedin.com/in/purbon/
> >
>


-- 
Santilli Jonathan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message