kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Palino <tpal...@gmail.com>
Subject Re: stunning error - Request of length 1550939497 is not valid, it is larger than the maximum size of 104857600 bytes
Date Tue, 14 Jul 2015 18:29:05 GMT
This is interesting. We have seen something similar internally at LinkedIn
with one particular topic (and Avro schema), and only once in a while.
We've seen it happen 2 or 3 times so far. We had chalked it up to bad
content in the message, figuring that the sender was doing something like
sending a long stream of a single character, in error, which was creating a
highly compressible message. Given these cases, I'm no longer certain
that's the case.

Becket, you had been taking a look at this internally. Do you have any
thoughts on this?

-Todd


On Tue, Jul 14, 2015 at 11:18 AM, JIEFU GONG <jgong@berkeley.edu> wrote:

> @Gwen
> I am having a very very similar issue where I am attempting to send a
> rather small message and it's blowing up on me (my specific error is:
> Invalid receive (size = 1347375956 larger than 104857600)). I tried to
> change the relevant settings but it seems that this particular request is
> of 1340 mbs (and davids will be 1500 mb) and attempting to change the
> setting will give you another error saying there is not enough memory in
> the java heap. Any insight here?
>
> Specifically I am speculating the issue is indeed what Shayne has said
> about encoding: I am trying to use apachebench to send a post request to a
> kafka server but it is returning the above error -- do I have to format the
> data in any way as this might be the reason why I'm experience this issue.
>
>
> On Sun, Jul 12, 2015 at 6:35 AM, Shayne S <shaynest113@gmail.com> wrote:
>
> > Your payload is so small that I suspect it's an encoding issue. Is your
> > producer set to expect a byte array and you're passing a string? Or vice
> > versa?
> >
> > On Sat, Jul 11, 2015 at 11:08 PM, David Montgomery <
> > davidmontgomery@gmail.com> wrote:
> >
> > > I cant send this soooo simple payload using python.
> > >
> > > topic: topic-test-development
> > > payload: {"utcdt": "2015-07-12T03:59:36", "ghznezzhmx": "apple"}
> > >
> > >
> > > No handlers could be found for logger "kafka.conn"
> > > Traceback (most recent call last):
> > >   File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line
> > 81,
> > > in <module>
> > >     test_send_data_to_realtimenode()
> > >   File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line
> > 38,
> > > in test_send_data_to_realtimenode
> > >     response = producer.send_messages(test_topic,test_payload)
> > >   File
> "/usr/local/lib/python2.7/dist-packages/kafka/producer/simple.py",
> > > line 54, in send_messages
> > >     topic, partition, *msg
> > >   File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> > > line 349, in send_messages
> > >     return self._send_messages(topic, partition, *msg)
> > >   File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> > > line 390, in _send_messages
> > >     fail_on_error=self.sync_fail_on_error
> > >   File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line
> > 480,
> > > in send_produce_request
> > >     (not fail_on_error or not self._raise_on_response_error(resp))]
> > >   File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line
> > 247,
> > > in _raise_on_response_error
> > >     raise resp
> > > kafka.common.FailedPayloadsError
> > >
> > > Here is what is in my logs
> > > [2015-07-12 03:29:58,103] INFO Closing socket connection to
> > > /xxx.xxx.xxx.xxx due to invalid request: Request of length 1550939497
> is
> > > not valid, it is larger than the maximum size of 104857600 bytes.
> > > (kafka.network.Processor)
> > >
> > >
> > >
> > > Server is 4 gigs of ram.
> > >
> > > I used export KAFKA_HEAP_OPTS=-Xmx256M -Xms128M in
> kafka-server-start.sh
> > >
> > > So.....why?
> > >
> >
>
>
>
> --
>
> Jiefu Gong
> University of California, Berkeley | Class of 2017
> B.A Computer Science | College of Letters and Sciences
>
> jgong@berkeley.edu <eliselim@berkeley.edu> | (925) 400-3427
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message