kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Reynolds <sreyno...@twilio.com.INVALID>
Subject Re: Default commit interval for Kafka Streams
Date Tue, 16 Apr 2019 19:43:24 GMT
So I think I understand. Restating your message and over simplifying it

With exactly once semantics, the downstream consumer must wait every
COMMIT_INTERVAL_MILLISECONDS for the producer to issue a commit prior
processing new records.

On Tue, Apr 16, 2019 at 10:39 AM Matthias J. Sax <matthias@confluent.io>
wrote:

> The commit interval is small to keep end-to-end processing latency
> small. For example, if data is repartitioned, a downstream task can only
> read the data after the upstream tasks commit its transaction.
>
> -Matthias
>
> On 4/16/19 9:56 AM, Scott Reynolds wrote:
> > Hi,
> >
> > I have been unable to determine  why the default commit interval for
> > Exactly Once Streams application is 100L. This seems really aggressive
> and
> > does produce a large amount of offsets to our broker.  I have changed
> this
> > in our application but I am worried I have now introduced a bug in the
> > application.
> >
> > Can some provide color and context into why we use 100L ms as the commit
> > interval for Exactly Once Streams api?
> >
> > 1.
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-129%3A+Streams+Exactly-Once+Semantics
> > 2. https://github.com/apache/kafka/pull/2945
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message