kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sybrandy, Casey" <Casey.Sybra...@Six3Systems.com>
Subject RE: Large message causing a MessageSizeTooLargeException
Date Thu, 16 Feb 2012 21:06:44 GMT
Nevermind.  I just realized when I started stripping my pom.xml to the bare necessities that
I forgot to upgrade the Kafka client.  Before, I only started the updated server.  Now I'm
getting other errors, but these appear to be with Zookeeper and potentially memory.

Thanks for your help.

Casey
________________________________________
From: Neha Narkhede [neha.narkhede@gmail.com]
Sent: Thursday, February 16, 2012 3:41 PM
To: kafka-users@incubator.apache.org
Subject: Re: Large message causing a MessageSizeTooLargeException

Casey,

This could be a bug. Do you mind filing a JIRA with a test case that
we can run to reproduce this problem ?

Thanks,
Neha

On Thu, Feb 16, 2012 at 12:24 PM, Sybrandy, Casey
<Casey.Sybrandy@six3systems.com> wrote:
> I'm not explicitly using an encoder.  All of my messages are encoded using Avro and I
send the resulting binary is passed to Kafka.  That binary array is definitely less than the
size of the buffer (About 37MB).  Checking the size of that was one of the first things I
did.
>
> This begs the question: is there a default encoder that's used?  If so, could that be
making the size larger than the buffer size?
>
> Casey
> ________________________________________
> From: Jun Rao [junrao@gmail.com]
> Sent: Thursday, February 16, 2012 2:02 PM
> To: kafka-users@incubator.apache.org
> Subject: Re: Large message causing a MessageSizeTooLargeException
>
> What encoder are you using? Is it possible that after encoding, the message
> size is larger than the threshold?
>
> Thanks,
>
> Jun
>
> On Thu, Feb 16, 2012 at 10:18 AM, Sybrandy, Casey <
> Casey.Sybrandy@six3systems.com> wrote:
>
>> Yes, I do still see the exception.  Is there a server setting that needs
>> to be updated?
>>
>> Has anyone else tried to push multi-megabyte files through Kafka?
>>
>> Casey
>> ________________________________________
>> From: Jun Rao [junrao@gmail.com]
>> Sent: Thursday, February 16, 2012 1:13 PM
>> To: kafka-users@incubator.apache.org
>> Subject: Re: Large message causing a MessageSizeTooLargeException
>>
>> There could be some transient failures in our unit tests especially on a
>> less powerful machine. Do you still see the exception?
>>
>> Thanks,
>>
>> Jun
>>
>> On Thu, Feb 16, 2012 at 9:22 AM, Sybrandy, Casey <
>> Casey.Sybrandy@six3systems.com> wrote:
>>
>> > Just tried it.  No change.  However, when I tried to create a release for
>> > it, it failed a number of unit tests.  I resorted to just packaging it
>> and
>> > starting zookeeper and kafka in that directory and appeared to work fine,
>> > though I'm unsure if the failed tests have anything to do with it.  I
>> don't
>> > suppose there's a pre-compiled release somewhere that I missed.
>> > ________________________________________
>> > From: Jun Rao [junrao@gmail.com]
>> > Sent: Thursday, February 16, 2012 11:35 AM
>> > To: kafka-users@incubator.apache.org
>> > Subject: Re: Large message causing a MessageSizeTooLargeException
>> >
>> > Could you try the 0.7 release, which is the first Apache release that we
>> > officially support?
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Thu, Feb 16, 2012 at 7:58 AM, Sybrandy, Casey <
>> > Casey.Sybrandy@six3systems.com> wrote:
>> >
>> > > No, I do not have compression enabled.
>> > >
>> > > Casey
>> > >
>> > > ________________________________________
>> > > From: Jun Rao [junrao@gmail.com]
>> > > Sent: Wednesday, February 15, 2012 6:31 PM
>> > > To: kafka-users@incubator.apache.org
>> > > Subject: Re: Large message causing a MessageSizeTooLargeException
>> > >
>> > > Did you enable compression? If so, try disable compression.
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > > On Wed, Feb 15, 2012 at 2:37 PM, Sybrandy, Casey <
>> > > Casey.Sybrandy@six3systems.com> wrote:
>> > >
>> > > > Hello,
>> > > >
>> > > > I have a weird bug that I'm trying to figure out.  I'm trying to
>> send a
>> > > > 30+ MB message through Kafka, but it keeps giving me a
>> > > > MessageSizeTooLargeException.  I did change the max.message.size
>> > property
>> > > > on the producer to be 50MB (50 * 1024 * 1024).  Is there something
>> I'm
>> > > > missing?  The consumer is also set to accept messages up to the same
>> > > size.
>> > > >
>> > > > Kafka 0.6.0
>> > > >
>> > > > Thanks.
>> > > >
>> > > > Casey
>> > >
>> >
>>

Mime
View raw message