kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghu Angadi <ang...@gmail.com>
Subject Re: relation between async mode and compression
Date Thu, 26 Apr 2012 03:20:55 GMT
will surely do that. Initially this will complement existing scribe
aggregators, and later might replace the aggregators.

thanks,
Raghu.

On Wed, Apr 25, 2012 at 7:08 PM, Jun Rao <junrao@gmail.com> wrote:

> Raghu,
>
> It's good to hear that Twitter will start to use Kafka in prod soon. Once
> that happens, do you think you can add your usage in the powerby wiki page?
>
> Thanks,
>
> Jun
>
> On Wed, Apr 25, 2012 at 6:31 PM, Raghu Angadi <raghuangadi@gmail.com>
> wrote:
>
> > another factor to be aware of is the effect of partitions. Even with
> async
> > producer in my use case, I saw lower compression ratio because the
> messages
> > are randomly split across 5 partitions.. A simple work around was to use
> a
> > key such that each of 1000 consecutive messages go to one partition.
> >
> > we are really happy with how well Kafka 'just worked' in our prototype at
> > twitter. will likely use it production soon..
> >
> > On Wed, Apr 25, 2012 at 10:47 AM, Felix GV <felix@mate1inc.com> wrote:
> >
> > > Also, compression ratios are usually better on larger payloads, so
> > > compression should, in most cases, be more effective when combined with
> > > async because the batching of messages results in larger payloads, and
> > thus
> > > better compression ratios.
> > >
> > > --
> > > Felix
> > >
> > >
> > >
> > > On Wed, Apr 25, 2012 at 8:14 AM, Joel Koshy <jjkoshy.w@gmail.com>
> wrote:
> > >
> > > > Hi Navneet,
> > > >
> > > > From docs, its not clear whether async mode and compression are
> related
> > > or
> > > > > independent. Can you please elaborate?
> > > > >
> > > >
> > > > They are related in the sense that the DefaultEventHandler that is
> used
> > > in
> > > > async mode compresses a batch of messages when they are serialized.
> > > > However, compression is not tied to async mode. Sync mode can also
> use
> > > > compression, but it does not do batching.
> > > >
> > > >
> > > > >
> > > > > One more thing - compression definitely enhances performance, but
> is
> > it
> > > > > true for async mode as well (assuming they are not related)?
> > > > >
> > > >
> > > > Async mode basically enables batching of messages so its throughput
> > gain
> > > is
> > > > orthogonal to that gained by compression.
> > > >
> > > > Thanks,
> > > >
> > > > Joel
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message