kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liam Clarke-Hutchinson <liam.cla...@adscale.co.nz>
Subject Re: Kafka Connect: producer idempotence after increasing producer.max.in.flight.requests.per.connection
Date Fri, 23 Apr 2021 06:12:29 GMT
Hi Yong Gang,

Idempotence is not guaranteed if you increase max in flight requests.
Because if 5 requests are dispatched, but 2 requests fail and are retried,
the messages in those 2 requests will arrive after the messages in the
other three requests, so ordering will be changed.

Assuming everything is happy in your cluster, network etc., you won't see
immediate negative effects on idempotence when increasing that property.

But when the cluster/network/etc. is heavily loaded, then you'll start to
see the negative effects.

Does that make sense?

Kind regards,

Liam Clarke-Hutchinson

On Fri, 23 Apr. 2021, 11:44 am Yong Gang, <che.yonggang@gmail.com> wrote:

> Hi guys,
>
> We are using Kafka Connect to do Kafka to Kafka data replication between
> DCs.
> For this config item *producer.max.in.flight.requests.per.connection* the
> default value in KafkaProducer (Kafka client) is 5 and can still
> maintain idempotence, but in Kafka Connect the default value of this is 1.
>
> I wonder if increasing *producer.max.in.flight.requests.per.connection *to
> 5 can Kafka Connect still be idempotent?
>
> Thanks,
> YongGang
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message