kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Zhou <frank.z...@itiviti.com>
Subject Message batch & compression doesn't work as expected
Date Thu, 12 Dec 2019 11:27:37 GMT
Hi,

I am testing kafka client on message batch and compression. I have enabled
message batching along with compression, with batch.size set to 3M,
linger.ms set to 5000ms and compression.type set to gzip(Attached whole
producer config). Then testing with wireshark to check the details.
First issue our team notice is that compression codec seems to have some
issue. Since we set it as gzip, but we notice in wireshark, it will display
as other compression codec, like Snappy in attached screenshot(Not sure if
this is wireshark's issue, or a real issue on Kafka, but the whole packet
details display seems fine in wireshark).
Second issue is that we have set the latency and batch number so high, but
it still send the Produce request to server much more frequently than we
expected. Size per message that is sending before batch & compression
should be around 200 bytes, and during testing, all the message generated
by us should be around 200KB, so we are expecting much less packets
transferred than this(screenshot only shows small amount of them, total
number is 1472).
[image: 2019-12-12_19h00_15.png]
 Is it we miss some config or the config is not correct leading to this?


-- 
*Frank Zhou*
R&D, Itiviti
Java Developer
D +852 2521 7480
frank.zhou@itiviti.com

______________________________

itiviti.com <https://www.itiviti.com/>

*The information contained in or attached to this email is strictly
confidential. If you are not the intended recipient, please notify us
immediately by telephone and return the message to us.*

*Email communications by definition contain personal information. The
Itiviti group of companies is subject to European data protection
regulations. Itiviti’s Privacy Notice is available at www.itiviti.com
<http://www.itiviti.com/>. Itiviti expects the recipient of this email to
be compliant with Itiviti’s Privacy Notice and applicable regulations.
Please advise us immediately at dataprotectionteam@Itiviti.com if you are
not compliant with these.*

-- 
______________________________

itiviti.com <https://www.itiviti.com/>
 
<https://www.linkedin.com/company/itiviti> Follow Itiviti on Linkedin 
<https://www.linkedin.com/company/10438325/>




The information contained 
in or attached to this email is strictly confidential. If you are not the 
intended recipient, please notify us immediately by telephone and return 
the message to us. Email communications by definition contain personal 
information. The Itiviti group of companies is subject to European data 
protection regulations.



Itiviti’s Privacy Notice is available at 
www.itiviti.com <http://www.itiviti.com/>. Itiviti expects the recipient of 
this email to be compliant with Itiviti’s Privacy Notice and applicable 
regulations. Please advise us immediately at dataprotectionteam@Itiviti.com 
if you are not compliant with these.

Mime
View raw message