kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "M. Manna" <manme...@gmail.com>
Subject Re: Message batch & compression doesn't work as expected
Date Thu, 12 Dec 2019 13:10:28 GMT
Frank,

On Thu, 12 Dec 2019 at 11:28, Frank Zhou <frank.zhou@itiviti.com> wrote:

> Hi,
>
> I am testing kafka client on message batch and compression. I have enabled
> message batching along with compression, with batch.size set to 3M,
> linger.ms set to 5000ms and compression.type set to gzip(Attached whole
> producer config). Then testing with wireshark to check the details.
> First issue our team notice is that compression codec seems to have some
> issue. Since we set it as gzip, but we notice in wireshark, it will display
> as other compression codec, like Snappy in attached screenshot(Not sure if
> this is wireshark's issue, or a real issue on Kafka, but the whole packet
> details display seems fine in wireshark).
> Second issue is that we have set the latency and batch number so high, but
> it still send the Produce request to server much more frequently than we
> expected. Size per message that is sending before batch & compression
> should be around 200 bytes, and during testing, all the message generated
> by us should be around 200KB, so we are expecting much less packets
> transferred than this(screenshot only shows small amount of them, total
> number is 1472).
> [image: 2019-12-12_19h00_15.png]
>  Is it we miss some config or the config is not correct leading to this?
>
>
>
I recently tuned our GCP based test cluster using batch of 800K, no
compression, and no linger.ms. We got the desired consistency and desired
throughput. But we used 2.3.0 version, and I don't suppose it matters much
at that point.

https://www.youtube.com/watch?v=oQe7PpDDdzA

The above shows a very good and detailed analysis done by Becket regarding
throughput calculation and estimation. Have you checked this to see what
matches your scenario?
The sending is impacted by linger.ms too, not just batch.size. So tuning
them together is a bit tricky. Perhaps you want to see which one you need
more.




> --
> *Frank Zhou*
> R&D, Itiviti
> Java Developer
> D +852 2521 7480
> frank.zhou@itiviti.com
>
> ______________________________
>
> itiviti.com <https://www.itiviti.com/>
>
> *The information contained in or attached to this email is strictly
> confidential. If you are not the intended recipient, please notify us
> immediately by telephone and return the message to us.*
>
> *Email communications by definition contain personal information. The
> Itiviti group of companies is subject to European data protection
> regulations. Itiviti’s Privacy Notice is available at www.itiviti.com
> <http://www.itiviti.com/>. Itiviti expects the recipient of this email to
> be compliant with Itiviti’s Privacy Notice and applicable regulations.
> Please advise us immediately at dataprotectionteam@Itiviti.com if you are
> not compliant with these.*
>
> ______________________________
>
> itiviti.com <https://www.itiviti.com/>
> <https://www.linkedin.com/company/itiviti> Follow Itiviti on Linkedin
> <https://www.linkedin.com/company/10438325/>
>
> The information contained in or attached to this email is strictly
> confidential. If you are not the intended recipient, please notify us
> immediately by telephone and return the message to us. Email communications
> by definition contain personal information. The Itiviti group of companies
> is subject to European data protection regulations.
>
> Itiviti’s Privacy Notice is available at www.itiviti.com. Itiviti expects
> the recipient of this email to be compliant with Itiviti’s Privacy Notice
> and applicable regulations. Please advise us immediately at
> dataprotectionteam@Itiviti.com if you are not compliant with these.
>

Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message