Hi,I am testing kafka client on message batch and compression. I have enabled message batching along with compression, with batch.size set to 3M, linger.ms set to 5000ms and compression.type set to gzip(Attached whole producer config). Then testing with wireshark to check the details.First issue our team notice is that compression codec seems to have some issue. Since we set it as gzip, but we notice in wireshark, it will display as other compression codec, like Snappy in attached screenshot(Not sure if this is wireshark's issue, or a real issue on Kafka, but the whole packet details display seems fine in wireshark).Second issue is that we have set the latency and batch number so high, but it still send the Produce request to server much more frequently than we expected. Size per message that is sending before batch & compression should be around 200 bytes, and during testing, all the message generated by us should be around 200KB, so we are expecting much less packets transferred than this(screenshot only shows small amount of them, total number is 1472).Is it we miss some config or the config is not correct leading to this?
https://www.youtube.com/watch?v=oQe7PpDDdzA
The above shows a very good and detailed analysis done by Becket regarding throughput calculation and estimation. Have you checked this to see what matches your scenario?
The sending is impacted by linger.ms too, not just batch.size. So tuning them together is a bit tricky. Perhaps you want to see which one you need more.
--Frank Zhou
R&D, Itiviti
Java Developer
D +852 2521 7480
frank.zhou@itiviti.com______________________________The information contained in or attached to this email is strictly confidential. If you are not the intended recipient, please notify us immediately by telephone and return the message to us.
Email communications by definition contain personal information. The Itiviti group of companies is subject to European data protection regulations. Itiviti’s Privacy Notice is available at www.itiviti.com. Itiviti expects the recipient of this email to be compliant with Itiviti’s Privacy Notice and applicable regulations. Please advise us immediately at dataprotectionteam@Itiviti.com if you are not compliant with these.
______________________________The information contained in or attached to this email is strictly confidential. If you are not the intended recipient, please notify us immediately by telephone and return the message to us. Email communications by definition contain personal information. The Itiviti group of companies is subject to European data protection regulations.
Itiviti’s Privacy Notice is available at www.itiviti.com. Itiviti expects the recipient of this email to be compliant with Itiviti’s Privacy Notice and applicable regulations. Please advise us immediately at dataprotectionteam@Itiviti.com if you are not compliant with these.