kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerrit Jansen van Vuuren <gerrit...@gmail.com>
Subject Re: Serious problem
Date Fri, 06 Jan 2017 19:36:49 GMT
The kafka brokers have a maximum message size limit, this is a protection
measure and avoids sending monster messages to kafka.

You have two options:
 1. On the brokers, increase the max.request.size, default is at ~2mb,
making it 5 or even 10 is not an issue normally. Java applications can
happily handle this size.
 2. breakup you're records into smaller size.

If you haven't enabled compression, enable compression in your client
configuration.

You should also probably inspect this particular message that goes over
2mb, if its web data sometimes bogus hrefs with looong query strings that
make no sense can be dropped in a pre processor before sending to kafka. If
its real custom data, you should try to rethink your data model and spend
some time on finding the right data format that provides best compression
and speed (e.g GPB, Avro, JSON + GZIP), csv is almost never what you want.

Hope this helps.

Regards,
 Gerrit


On Fri, Jan 6, 2017 at 8:12 AM, zhilong wang <wangzhilong0905@gmail.com>
wrote:

> flume-kafka-sink send message to kafka-0.9
>
>
>
>
> 06 Jan 2017 13:13:36,595 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.SinkRunner$PollingRunner.run:158)  - Unable to deliver
> event. Exception follows
> org.apache.flume.EventDeliveryException: Failed to publish events
> at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:252)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(
> DefaultSinkProcessor.java:67)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.RecordTooLargeException: The message is
> 2742363 bytes when serialized which is larger than th
> e maximum request size you have configured with the max.request.size
> configuration.
> at
> org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(
> KafkaProducer.java:686)
> at
> org.apache.kafka.clients.producer.KafkaProducer.send(
> KafkaProducer.java:449)
> at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:212)
> ... 3 more
> Caused by: org.apache.kafka.common.errors.RecordTooLargeException: The
> message is 2742363 bytes when serialized which is larger than the maximum
> request size you have configure
> d with the max.request.size configuration.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message