kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitry Minkovsky <dminkov...@gmail.com>
Subject Re: High end-to-end latency with processing.guarantee=exactly_once
Date Thu, 20 Dec 2018 02:12:37 GMT
Hello 王美功,

I am using 2.1.0. And, I think you nailed it on the head, because my
application is low throughput and I am seeing UNKNOWN_PRODUCER_ID all the
time with exactly once enabled. I've googled this before but couldn't
identify the cause. Thank you!

Setting retry.backoff.ms to 5 brought the latency down from 1.3s to 750ms.
That's tolerable for me, but I am wondering: when the KAFKA-7190 is fixed,
will the latency drop further?

Dmitry

On Wed, Dec 19, 2018 at 8:38 PM meigong.wang <meigong.wang@okcoin.com>
wrote:

> Which version are you using? This bug(
> https://issues.apache.org/jira/browse/KAFKA-7190) may increase the
> latency of your application, try to reduce the retry.backoff.ms,the
> default value is 100 ms.
>
>
> 王美功
>
>
> 原始邮件
> 发件人:Dmitry Minkovskydminkovsky@gmail.com
> 收件人:usersusers@kafka.apache.org
> 发送时间:2018年12月20日(周四) 09:25
> 主题:High end-to-end latency with processing.guarantee=exactly_once
>
>
> I have a process that spans several Kafka Streams applications. With the
> streams commit interval and producer linger both set to 5ms, when exactly
> once delivery is disabled, this process takes ~250ms. With exactly once
> enabled, the same process takes anywhere from 800-1200ms. In Enabling
> Exactly-Once in Kafka Streams ,">
> https://www.confluent.io/blog/enabling-exactly-kafka-streams/, Guozhang
> writes In Kafka Streams, because a new transaction is created whenever
> commit is called, the average transaction size is determined by the commit
> interval: with the same incoming traffic, a shorter commit interval will
> result in smaller transactions. In practice users should therefore tune the
> commit.interval.ms setting when exactly-once is enabled to make a good
> trade-off between throughput versus end-to-end processing latency. But I am
> not seeing much of a difference when I tune commit.interval.ms with
> exactly once enabled. `though()` and `.to()/.stream()` take 100-250ms even
> with commit.interval.ms set to 5ms. Do these latency differences sound
> right? Is something off? Thank you, Dmitry

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message