kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Smriti Jha <smr...@agolo.com>
Subject Kafka producer drops large messages
Date Tue, 11 Apr 2017 18:53:54 GMT
Hello all,

Can somebody shed light on kafka producer's behavior when the total size of
all messages in the buffer (bounded by queue.buffering.max.ms) exceeds the
socket buffer size (send.buffer.bytes)?

I'm using Kafka v0.8.2 with the old Producer API and have noticed that our
systems are dropping a few messages that are closer to 1MB in size. A few
messages that are only a few KBs in size and are attempted to be sent
around the same time as >1MB messages also get dropped. The official
documentation does talk about never dropping a "send" in case the buffer
has reached queue.buffering.max.messages but I don't think that applies to
size of the messages.

Thanks!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message