kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ofir Sharony <ofir.shar...@myheritage.com>
Subject Re: Unable to consume compressed topics
Date Thu, 22 Dec 2016 07:27:08 GMT
Yes,
The console consumer works great, on all compression types.

*Ofir Sharony*
BackEnd Tech Lead

Mobile: +972-54-7560277 | ofir.sharony@myheritage.com | www.myheritage.com
MyHeritage Ltd., 3 Ariel Sharon St., Or Yehuda 60250, Israel

<http://www.myheritage.com/>

<https://www.facebook.com/myheritage>
<https://twitter.com/myheritage>         <http://blog.myheritage.com/>
    <https://www.youtube.com/user/MyHeritageLtd>


On Wed, Dec 21, 2016 at 6:07 PM, Ismael Juma <ismael@juma.me.uk> wrote:

> Does it work if you use the console consumer from 0.10.1 and/or 0.10.0?
>
> Ismael
>
> On Wed, Dec 21, 2016 at 5:06 AM, Ofir Sharony <ofir.sharony@myheritage.com
> >
> wrote:
>
> > Hi guys,
> >
> > I'm trying to consume our Kafka topics using Kafkacat
> > <https://github.com/edenhill/kafkacat>.
> > We were able to successfully consume messages using the the default
> > compression.type (producer, i.e without compression in our case).
> > When we changed the compression type to either compression algorithm
> (gzip
> > / snappy / lz4), the consumption fails immediately with a fatal error.
> >
> > Here's the command:
> >
> > kafkacat -X api.version.request=true -b localhost -t <topic> -o end -C
> -c 1
> > -v -p 4
> >
> > And the response received when consuming a topic configured with any of
> the
> > compression algorithms:
> >
> > % Fatal error at consume_cb:407:
> >
> > % ERROR: Topic maxwell_with_request_site_sharded [4] error: Message at
> > offset 733099 is too large to fetch, try increasing
> > receive.message.max.bytes
> >
> > The response is the same for all offsets, in different partitions and all
> > our topics.
> > Please notice the messages size is quite small (smaller than the value of
> > receive.message.max.bytes),
> > I'm using the latest Kafka version (0.10.1).
> >
> > Please advise.
> > Thanks,
> >
> > *Ofir Sharony*
> > BackEnd Tech Lead
> >
> > Mobile: +972-54-7560277 <+972%2054-756-0277> |
> ofir.sharony@myheritage.com
> > | www.myheritage.com
> > MyHeritage Ltd., 3 Ariel Sharon St., Or Yehuda 60250, Israel
> >
> > <http://www.myheritage.com/>
> >
> > <https://www.facebook.com/myheritage>
> > <https://twitter.com/myheritage>         <http://blog.myheritage.com/>
> >     <https://www.youtube.com/user/MyHeritageLtd>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message