kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Schofield <andrew_schofield_j...@outlook.com>
Subject RE: Exactly-once publication behaviour
Date Sun, 21 Feb 2016 13:52:15 GMT
Hi,
That's good, but somewhat mysterious :) I'd like to help accelerate exactly-one behaviour.
 Perhaps it might be a candidate for the release after 0.10, if there's sufficient interest
in the community.

Could someone share the latest thinking on this subject? I'm not sure what the appropriate
forum is. How have you previously been sharing high-level architectural plans for people to
pick up and deliver? Since there's no KIP I guess it's a bit premature to discuss on the KIP
call.

Thanks
Andrew

> Subject: Re: Exactly-once publication behaviour
> From: ben@confluent.io
> Date: Fri, 19 Feb 2016 15:13:07 -0800
> To: users@kafka.apache.org
> 
> Hi Andrew
> 
> There are plans to add exactly once behaviour. This will likely be a little more than
Idempotent producers with the motivation being to provide better delivery guarantees for Connect,
Streams and Mirror Maker. 
> 
> B
> 
> 
> 
>> On 19 Feb 2016, at 13:54, Andrew Schofield  wrote:
>> 
>> When publishing messages to Kafka, you make a choice between at-most-once and at-least-once
delivery, depending on whether you wait for acknowledgments and whether you retry on failures.
In most cases, those options are good enough. However, some systems offer exactly-once reliability
too. Although my view is that the practical use of exactly-once is limited in the situations
that Kafka is generally used for, when you're connecting other systems to Kafka or bridging
between protocols, I think there is value in propagating the reliability level that the other
system expects.
>> 
>> As a consumer, you can manage your offset and get exactly-once delivery, or more
likely exactly-once processing, of the messages.
>> 
>> I've read about idempotent producers (https://cwiki.apache.org/confluence/display/KAFKA/Idempotent+Producer)
and I know there's been some discussion about transactions too.
>> 
>> Is there a plan to provide the tools to enable exactly-once publication behaviour?
Is this a planned enhancement to Kafka Connect? Is there already some technique that people
are using effectively to get exactly-once?
>> 
>> Andrew Schofield 
> 
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message