kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Inder Pall <inder.p...@gmail.com>
Subject Re: Compression
Date Thu, 08 Dec 2011 05:24:44 GMT
Thanks Neha. it's clear now.



On Wed, Dec 7, 2011 at 11:31 PM, Neha Narkhede <neha.narkhede@gmail.com>wrote:

> Inder,
>
> Sorry for not being clear. Compression is done at the producer side only.
> The producer takes a batch of messages, compresses it into a message and
> sends it to the broker.
> Nothing changes on the broker side. It simply writes that compressed
> message to the partition.
>
> The consumer reads a set of messages, detects the ones that are compressed
> messages, and transparently decompresses them before handing it over to the
> end user.
>
> Take a look at the CompressionUtils class and see how it is used.
>
> Thanks,
> Neha
>
> On Wed, Dec 7, 2011 at 9:41 AM, Inder Pall <inder.pall@gmail.com> wrote:
>
> > Neha,
> >
> > to ensure i get it right, how would this work...messages would be coming
> > from producer
> > and let's say there isn't any batching so they become available to the
> > consumer almost realtime.
> > Now does since kafka has to retain them till the retention period broker
> > partitions would be compressed.
> >
> >
> >
> > On Wed, Dec 7, 2011 at 10:03 PM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > Inder,
> > >
> > > Compression in Kafka is implemented at message set level. This means
> that
> > > it compresses n messages and represents that as one message.
> > > So for a compression-enabled topic, in the broker partition, each
> message
> > > has several messages wrapped inside.
> > >
> > > It is true that compression is most effective on very large block
> sizes,
> > > like a MB or so. But even with message sets of size 40-50K, we see a
> > > compression ratio of ~3x on our data.
> > > This is using GZIP, you may see different results with Snappy.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Wed, Dec 7, 2011 at 8:29 AM, Inder Pall <inder.pall@gmail.com>
> wrote:
> > >
> > > > Folks,
> > > >
> > > > may be this has been asked/discussed before and i missed it. How do
> we
> > do
> > > > compression in kafka (at record level)?
> > > > So it would like each line/record in the broker partiton would be
> > > > compressed. But our individual records aren't big in size do we
> really
> > > see
> > > > the value of compression
> > > >
> > > > --
> > > > Thanks,
> > > > - Inder
> > > >  Tech Platforms @Inmobi
> > > >  Linkedin - http://goo.gl/eR4Ub
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > - Inder
> >  Tech Platforms @Inmobi
> >  Linkedin - http://goo.gl/eR4Ub
> >
>



-- 
Thanks,
- Inder
  Tech Platforms @Inmobi
  Linkedin - http://goo.gl/eR4Ub

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message