kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Coleman <John...@central.tech>
Subject Re: High latency for Kafka transactions
Date Thu, 03 Dec 2020 03:57:27 GMT
Hi,

We send small transactions (2 to 3 messages) a few times per second from each of 5 to 10 producers,
depending on the consumer partition size.

Regards,
John

On 14/11/20 03:55, "Guozhang Wang" <wangguoz@gmail.com> wrote:

    Hello John,

    It's a bit hard to reason what's your total producing traffic to the Kafka
    brokers, e.g. how frequently do you send a `whole group of messages`? If it
    is every 10ms e.g. then your traffic would be 100K per sec, which is pretty
    large enough with transactions.


    Guozhang

    On Fri, Nov 13, 2020 at 1:09 AM John Coleman <John.co@central.tech> wrote:

    > Hi,
    >
    > We have very high latency ~1 second when we send a small number of small
    > messages to Kafka. The whole group of messages is less than 1k. We are
    > thinking better to just send the messages together as 1 message containing
    > a collection of all the messages.
    >
    > Is there something that we can tweak to reduce this latency? We have 10
    > producers and they publish to a topic with 5 consumers. Be nice if we don’t
    > have to change our message structure.
    >
    > TIA
    > John
    >


    -- 
    -- Guozhang

Mime
View raw message