kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörg Wagner <joerg.wagn...@1und1.de>
Subject Compression and MirrorMaker
Date Tue, 01 Sep 2015 07:33:58 GMT

I was looking into compression for some WAN mirroring and did some 

Our producer is using snappy and I think I can see that in the network 
traffic. While the outer part of each message is readable, the content 
looks compressed to me with mostly nonprintable characters. However upon 
consuming by the MirrorMaker complete message contents are readable in 
the dump. To clarify:

producer --(snappy)--> broker --(uncompressed, WAN)--> MirrorMaker(consumer)

 From what I read about kafka (end-to-end) compression that shouldn't 
even be possible? As far as I can see compression can only be configured 
at the producer.
How can I be sure especially the communication via WAN is compressed?
Does anybody have experience mirroring via WAN and can give some hints 
on the MM configuration as well as OS Network tuning regarding kafka?

Also the option shallow.iterator.enable does not seem to be documented. 
Regarding the above, could that help (although I don't suppose so, 
because the messages already have to leave the broker compressed)?


View raw message