kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Fwd: Unexpected end of ZLIB input stream
Date Sat, 01 Dec 2012 02:25:08 GMT
Dmitri,

Snappy tends to be much faster than gzip with a slightly lower compression
ratio. Not sure how stable it is though.

Thanks,

Jun

On Fri, Nov 30, 2012 at 1:33 PM, Dmitri Priimak <
priimak@highwire.stanford.edu> wrote:

> Well, the files that I am sending are themselves not compressed, but I do
> use gzip compression when
> sending data. kafka client is configured with option
>
>     compression.codec=1
>
> which implies gzip compression. Setting that value to 2 means "Snappy
> compression". What compression
> would you recommend? Is "Snappy compression" any good?
>
> --
> Dmitri Priimak
>
> On 11/30/2012 01:17 PM, Paul Sutter wrote:
> > Any chance the stream was created by the gzip command? If so, that just
> happens sometime. The only
> > way we found is to confirm that the error happened at the expected end
> of the file and not a
> > too-short file (we learned this dealing with petabytes of gziped
> logfiles)
> >
> > Apologies if thats irrelevant.
> >
> > Begin forwarded message:
> >
> >> *From:* Dmitri Priimak <priimak@highwire.stanford.edu <mailto:
> priimak@highwire.stanford.edu>>
> >> *Date:* November 30, 2012 12:26:10 PM PST
> >> *To:* users@kafka.apache.org <mailto:users@kafka.apache.org>
> >> *Subject:* *Re: Unexpected end of ZLIB input stream*
> >> *Reply-To:* users@kafka.apache.org <mailto:users@kafka.apache.org>
> >>
> >> Well, it does not look like I can reproduce it anymore. While I was
> able to consistently crash it
> >> yesterday, today kafka broker runs without any problems.
> >>
> >> --
> >> Dmitri Priimak
> >>
> >> On 11/29/2012 02:10 PM, Dmitri Priimak wrote:
> >>> On 11/29/2012 02:06 PM, Neha Narkhede wrote:
> >>>> That just meant that we couldn't reproduce it during our testing, but
> >>>> we occasionally do see it in production, which is much harder to
> >>>> reproduce.
> >>>> It will be great if you can reproduce the issue and attach the test
> case.
> >>> I see. I will try to isolate this problem. I am hoping that I can
> locate exact sequence of messages
> >>> that trigger this problem, but at the moment I cannot easily do that.
> >>>
> >>> --
> >>> Dmitri Priimak
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message