kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Errors in load tests
Date Sat, 10 Dec 2011 00:33:06 GMT
Florian,

Could you try the 0.7 branch in Apache? We have fixed some consumer-side
bugs that can cause InvalidMessageSize issue between 0.6 and 0.7.

Thanks,

Jun

On Fri, Dec 9, 2011 at 1:12 PM, Florian Leibert <flo@leibert.de> wrote:

> My observations so far:
>
> If I start with 1 broker, 1 consumer (code is pretty much the same as in
> the simple-consumer-demo) and 1 producer I can do arbitrary many runs and
> everything works fine. Adding a broker after the producer stopped and
> restarting the producer (consumer still running) throws on the consumer
> side the InvalidMessageSize exception - which continues to happen on
> subsequent produce/consume cycles as long as the second broker is running.
>
> I looked around Zookeeper - there are 8 partitions when one broker is
> active and 16 with two... i see all the topics are registered in zookeeper
> and overall zookeeper logs are clean.
>
> After this fact I tried to see if two brokers and one partition each would
> work so I shutdown the brokers, changed the global partition size to 1 in
> each of the configs and restarted them. The result is the same, I still get
> InvalidMessageSize exceptions on the consumer size.
>
> On Fri, Dec 9, 2011 at 11:10 AM, Florian Leibert <flo@leibert.de> wrote:
>
> > On that note - another error I see right around the same time is below -
> i
> > see it on the consumer end. Note that in this case I have one consumer
> with
> > a single iterator - and on each broker 8 partitions. When printing
> > messages, I see an empty message on occasion...
> >
> > Exception in thread "main" kafka.common.InvalidMessageSizeException
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> >  at
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > at java.lang.Class.newInstance0(Class.java:355)
> >  at java.lang.Class.newInstance(Class.java:308)
> > at kafka.common.ErrorMapping$.maybeThrowException(ErrorMapping.scala:53)
> >  at
> >
> kafka.message.ByteBufferMessageSet.iterator(ByteBufferMessageSet.scala:60)
> > at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:67)
> >  at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:30)
> > at
> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:58)
> >  at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:50)
> > at com.airbnb.logging.util.ConsumerTest.main(ConsumerTest.java:50)
> >  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >  at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> >  at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
> >
> >
>
>
> --
> Best regards,
>
> Florian
> http://twitter.com/flo <http://twitter.com/floleibert>
> http://flori.posterous.com/
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message