kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liam Clarke-Hutchinson <liam.cla...@adscale.co.nz>
Subject Re: High latency for Kafka transactions
Date Thu, 03 Dec 2020 04:10:45 GMT
Hi John,

The default for the C# client is actually 5ms, not that it really makes a
difference. Are you able to share any of your producer code, especially
around the configuration?

Kind regards,

Liam Clarke

On Thu, Dec 3, 2020 at 5:01 PM John Coleman <John.co@central.tech> wrote:

> Hi,
>
> We are using Confluent.Kafka (dotnet) and according to the docs the
> linger.ms is default to 0.5ms which would cover a lot of serializations
> since each occurs in < 10 microseconds. A typical transaction batch would
> be 2 to 3 messages, occasionally more.
>
> Regards,
> John
>
> On 14/11/20 05:24, "Gary Russell" <grussell@vmware.com> wrote:
>
>     Have you increased the producer linger.ms property [1] to ~1s?
> (Default is 0 so records are sent immediately; it generally shouldn't be
> more than a few ms - depending on how long your serialization takes).
>
>     If not, perhaps your serialization (or compression) might be the
> problem; best to profile the app to see where the time is being spent.
>
>     [1]:
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkafka.apache.org%2Fdocumentation%2F%23linger.ms&amp;data=04%7C01%7CJohn.co%40central.tech%7Cee6e35911f484f03184208d88822d5e0%7C817e531d191b4cf58812f0061d89b53d%7C0%7C0%7C637409030504161229%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=t6XTDwN4QQs4kL5pMZzCSpcbjwMX95QrA8dzpDdrjyo%3D&amp;reserved=0
>     ________________________________
>     From: John Coleman <John.co@central.tech>
>     Sent: Friday, November 13, 2020 4:09 AM
>     To: users@kafka.apache.org <users@kafka.apache.org>
>     Subject: High latency for Kafka transactions
>
>     Hi,
>
>     We have very high latency ~1 second when we send a small number of
> small messages to Kafka. The whole group of messages is less than 1k. We
> are thinking better to just send the messages together as 1 message
> containing a collection of all the messages.
>
>     Is there something that we can tweak to reduce this latency? We have
> 10 producers and they publish to a topic with 5 consumers. Be nice if we
> don’t have to change our message structure.
>
>     TIA
>     John
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message