kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Devendra Tagare <devendra.tag...@pubmatic.com>
Subject Issues with Kafka async producer's enqueue timeout
Date Thu, 20 Nov 2014 13:01:08 GMT
Hi,

We are using an async producer to send data to a kafka cluster.The event rate at peak is around
250 events/second of size 25KB each.

In the producer code base we have added specific debug statements to capture the time taken
to create a producer,create a keyed message with a byte payload & send the message.

We have added the below properties to the ProducerConfig

queue.enqueue.timeout.ms=20
send.buffer.bytes=1024000
topic.metadata.refresh.interval.ms=30000

We also checked if these properties are being picked.

Based on the documentation, producer.send() queues the message on the async producer's queue.

So, ideally if the queue is full then the enqueue operation should result in an kafka.common.QueueFullException
in 20 ms.

The logs indicate that the enqueue operation is taking more than 20ms (takes around 250ms)
without throwing any exceptions.

Is there any other property that could conflict with queue.enqueue.timeout.ms which is causing
this behavior ?

Or is it possible that the queue is not full & yet the producer.send() call is still taking
around 200ms under peak load ?

Also, could you suggest any other alternatives so that we can either enforce a timeout or
throw an exception in-case the async producer is taking more than a specified amount of time
to queue an event.


Regards,
Dev

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message