kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitry Minkovsky <dminkov...@gmail.com>
Subject High end-to-end latency with processing.guarantee=exactly_once
Date Thu, 20 Dec 2018 01:25:08 GMT
I have a process that spans several Kafka Streams applications. With the
streams commit interval and producer linger both set to 5ms, when exactly
once delivery is disabled, this process takes ~250ms. With exactly once
enabled, the same process takes anywhere from 800-1200ms.

In Enabling Exactly-Once in Kafka Streams
<https://www.confluent.io/blog/enabling-exactly-kafka-streams/>, Guozhang

In Kafka Streams, because a new transaction is created whenever commit is
called, the average transaction size is determined by the commit interval:
with the same incoming traffic, a shorter commit interval will result in
smaller transactions. In practice users should therefore tune the
commit.interval.ms setting when exactly-once is enabled to make a good
trade-off between throughput versus end-to-end processing latency.

But I am not seeing much of a difference when I tune commit.interval.ms
with exactly once enabled. `though()` and `.to()/.stream()` take 100-250ms
even with commit.interval.ms set to 5ms.

Do these latency differences sound right? Is something off?

Thank you,

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message