kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manikumar <manikumar.re...@gmail.com>
Subject Re: is a topic compressed?
Date Thu, 21 Sep 2017 08:32:49 GMT
you can try DumpLogSegments tools to verify messages from log files.  This
will give compression type for each message.
https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-
DumpLogSegment

On Thu, Sep 21, 2017 at 1:38 PM, Vincent Dautremont <
vincent.dautremont@olamobile.com.invalid> wrote:

> Hi,
> Snappy keeps a lot of parts in plain text :
> look that example where only "pedia" is encoded/tokenized in the sentence.
> https://en.wikipedia.org/wiki/Snappy_(compression)
>
> > Wikipedia is a free, web-based, collaborative, multilingual encyclopedia
> > project.
>
>
> your data is then probably compressed with snappy.
>
> Another try would be to change compression to other values (or remove
> compression) and compare the tcp dump with the one you already have.
>
>
> Vincent.
>
> On Wed, Sep 20, 2017 at 11:58 PM, Wesley Chow <wes@chartbeat.com> wrote:
>
> > I have a producer configured to snappy compress data sent to a cluster.
> Is
> > there some way to verify that the data indeed is being compressed? If I
> > peek at the .log files on the broker, I can read some plain text amongst
> > binary. Similarly, tcpdump shows plain text readable data. I do not know
> if
> > this is evidence that compression is not working, but is there a better
> way
> > to verify that compression is turned on?
> >
> > Wes
> >
>
> --
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission, dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from any
> computer.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message