kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Loddengaard <a...@confluent.io>
Subject Re: kafka.common.QueueFullException
Date Thu, 18 Feb 2016 17:38:56 GMT
Hi Saurabh,

This is occurring because the produce message queue is full when a produce
request is made. The size of the queue is configured
via queue.buffering.max.messages. You can experiment with increasing this
(which will require more JVM heap space), or fiddling with
queue.enqueue.timeout.ms to control the blocking behavior when the queue is
full. Both of these configuration options are explained here:

https://kafka.apache.org/08/configuration.html

I didn't quite follow your last paragraph, so I'm not sure if the following
advice is applicable to you or not. You may also experiment with adding
more producers (either on the same or different machines).

I hope this helps.

Alex

On Thu, Feb 18, 2016 at 2:12 AM, Saurabh Kumar <saurabh.85@gmail.com> wrote:

> Hi,
>
> We have a Kafka server deployment shared between multiple teams and i have
> created a topic with multiple partitions on it for pushing some JSON data.
>
> We have multiple such Kafka producers running from different machines which
> produce/push data to a Kafka topic. A lot of times i see the following
> exception in the logs : "*Event queue is full of unsent messages, could not
> send event"*
>
> Any idea how to solve this ? We can not synchronise the volume or timing of
> different Kafka producers across machines and between multiple processes.
> There is a limit on maximum number of concurrent processes (kafka producer)
>  that can run on a mchine but it is only going to increase with time as we
> scale. What is the right way to solve this problem ?
>
> Thanks,
> Saurabh
>



-- 
*Alex Loddengaard | **Solutions Architect | Confluent*
*Download Apache Kafka and Confluent Platform: www.confluent.io/download
<http://www.confluent.io/download>*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message