kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Inder Pall <inder.p...@gmail.com>
Subject Re: Compression
Date Wed, 07 Dec 2011 17:41:56 GMT
Neha,

to ensure i get it right, how would this work...messages would be coming
from producer
and let's say there isn't any batching so they become available to the
consumer almost realtime.
Now does since kafka has to retain them till the retention period broker
partitions would be compressed.



On Wed, Dec 7, 2011 at 10:03 PM, Neha Narkhede <neha.narkhede@gmail.com>wrote:

> Inder,
>
> Compression in Kafka is implemented at message set level. This means that
> it compresses n messages and represents that as one message.
> So for a compression-enabled topic, in the broker partition, each message
> has several messages wrapped inside.
>
> It is true that compression is most effective on very large block sizes,
> like a MB or so. But even with message sets of size 40-50K, we see a
> compression ratio of ~3x on our data.
> This is using GZIP, you may see different results with Snappy.
>
> Thanks,
> Neha
>
>
> On Wed, Dec 7, 2011 at 8:29 AM, Inder Pall <inder.pall@gmail.com> wrote:
>
> > Folks,
> >
> > may be this has been asked/discussed before and i missed it. How do we do
> > compression in kafka (at record level)?
> > So it would like each line/record in the broker partiton would be
> > compressed. But our individual records aren't big in size do we really
> see
> > the value of compression
> >
> > --
> > Thanks,
> > - Inder
> >  Tech Platforms @Inmobi
> >  Linkedin - http://goo.gl/eR4Ub
> >
>



-- 
Thanks,
- Inder
  Tech Platforms @Inmobi
  Linkedin - http://goo.gl/eR4Ub

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message