kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Santilli <jonathansanti...@gmail.com>
Subject Re: RecordTooLargeException on 16M messages in Kafka?
Date Fri, 16 Aug 2019 08:38:08 GMT
Ok, it means that, using the KafkaProducer class setting the config
max.request.size worked correctly, right?
Now you are wondering about the kafka-console-producer.sh utility, right?
if so, have you tried using the parameter:

--producer.config /path/to/the/config/producer.properties


Just let me know, cheers!
--
Jonathan




On Thu, Aug 15, 2019 at 8:09 PM l vic <lvic4594@gmail.com> wrote:

> setting max.request.size worked when i did it in producer code; don't
> understand what producer.properties is for, seems like it's not used.
>
> On Thu, Aug 15, 2019 at 2:25 PM Jonathan Santilli <
> jonathansantilli@gmail.com> wrote:
>
> > I have asked because I did not see that in your previous email when you
> > tried the console producer.
> >
> > Jonathan.
> >
> > On Thu, Aug 15, 2019, 3:07 PM l vic <lvic4594@gmail.com> wrote:
> >
> > > yes, in producer.properties
> > >
> > > On Thu, Aug 15, 2019 at 9:59 AM Jonathan Santilli <
> > > jonathansantilli@gmail.com> wrote:
> > >
> > > > Just to be sure, please confirm the configuration parameter is well
> > > > set/configure at producer level:
> > > >
> > > > max.request.size = 177772390 (for instance)
> > > >
> > > > Cheers!
> > > > --
> > > > Jonathan
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Aug 15, 2019 at 1:44 PM l vic <lvic4594@gmail.com> wrote:
> > > >
> > > > > I tested it with kafka-console-consumer and kafka-console-producer
> > > > reading
> > > > > from large text file (no newlines):
> > > > >
> > > > > kafka-console-producer.sh --broker-list 10.10.105.24:6667
>  --topic
> > > > test <
> > > > > ./large-file
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Aug 15, 2019 at 4:49 AM Jonathan Santilli <
> > > > > jonathansantilli@gmail.com> wrote:
> > > > >
> > > > > > Hello, try to send and flush just one message of 16777239 bytes,
> to
> > > > > verify
> > > > > > the error still shows up.
> > > > > >
> > > > > > Cheers!
> > > > > > --
> > > > > > Jonathan
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Aug 15, 2019 at 2:23 AM l vic <lvic4594@gmail.com>
> wrote:
> > > > > >
> > > > > > > My kafka (1.0.0) producer errors out on  large (16M) messages.
> > > > > > > ERROR Error when sending message to topic test with key:
null,
> > > value:
> > > > > > > 16777239 bytes with error:
> > > > > (org.apache.kafka.clients.producer.internals.
> > > > > > > ErrorLoggingCallback)
> > > > > > >
> > > > > > > org.apache.kafka.common.errors.RecordTooLargeException:
The
> > message
> > > > is
> > > > > > > 16777327 bytes when serialized which is larger than the
maximum
> > > > request
> > > > > > > size you have configured with the max.request.size
> configuration.
> > > > > > > I found couple of links describing the solution:
> > > > > > > *
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> > > > > > > >*
> > > > > > >
> > > > > > > in my server.properties on brokers I set:
> > > > > > > socket.request.max.bytes=104857600
> > > > > > > message.max.bytes=18874368
> > > > > > > max.request.size=18874368
> > > > > > > replica.fetch.max.bytes=18874368
> > > > > > > fetch.message.max.bytes=18874368
> > > > > > >
> > > > > > > Then in my producer.properties i tried to set
> > > > > > > max.request.size=18874368
> > > > > > >
> > > > > > > But no matter how i large i try to set max.request.size
-
> > > > > > > i still have the same problem...Are there other settings
i am
> > > > missing?
> > > > > > > Can it be solved in configuration alone, or do i need to
make
> > code
> > > > > > changes?
> > > > > > > Thank you,
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Santilli Jonathan
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Santilli Jonathan
> > > >
> > >
> >
>


-- 
Santilli Jonathan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message