kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hogue <csho...@gmail.com>
Subject EventHandler in 0.8
Date Fri, 02 Aug 2013 15:26:36 GMT
We're a heavy 0.7 user and are now digging into 0.8 for some new projects.
One of the features we used in 0.7 appears to be different and not clearly
supported in 0.8.

We use the EventHandler plug-point in 0.7, specifically to do custom
batching before the messages are actually sent to the broker. We were
hoping to use this feature in 0.8, albeit for a slightly different reason.
In testing 0.8 we found the broker had a pretty significant choke point
with compression. I'm sure this is a known performance consideration, where
using the built-in compression forces the broker to un-compress and
re-compress the messages to assign offsets.

We changed our test to batch and compress the messages before it went to
the producer, leaving the compression.codec set to none. With this change
we saw on the order of a 3x increase in throughput on the broker while
still retaining compressed messages across the wire and on disk.

This was good for the tests but in the app we are planning to use semantic
partitioning, so pre-batching and compressing before going to the producer
is trickier. The hope was to use the producer's batching and partition
selection, just plugging into the event handler for the compression piece.
While I see the DefaultEventHandler is there and potentially replaceable
via the constructor, it appears to have taken on more functionality and
would thus be more involved to replace.

Combined with all produce requests across brokers now being on a single
thread, this is appearing to be difficult to achieve.

I can imagine why some of these changes are in place in light of the
replication feature, but wanted to throw it out there (a) in case we missed
something or (b) at least as a potential consideration in the client
re-design project that is under way.

I'd be happy to have any input or other ideas on how to accomplish this
that we may have missed. We'll look at other ways to tackle this and follow
the other threads about the new clients in the meantime.



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message