kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Devendra Tagare <devendra.tag...@pubmatic.com>
Subject RE: Issues with Kafka async producer's enqueue timeout
Date Fri, 21 Nov 2014 05:56:40 GMT
Hi,

We are using an async producer.

The producer properties are:

producer.type= async
queue.enqueue.timeout.ms=20
send.buffer.bytes=1024000
topic.metadata.refresh.interval.ms=30000


Please find more details in the mail thread below.


Regards,
Dev
________________________________
From: Devendra Tagare
Sent: Thursday, November 20, 2014 6:31 PM
To: users@kafka.apache.org
Cc: Devendra Tagare
Subject: Issues with Kafka async producer's enqueue timeout

Hi,

We are using an async producer to send data to a kafka cluster.The event rate at peak is around
250 events/second of size 25KB each.

In the producer code base we have added specific debug statements to capture the time taken
to create a producer,create a keyed message with a byte payload & send the message.

We have added the below properties to the ProducerConfig

producer.type= async
queue.enqueue.timeout.ms=20
send.buffer.bytes=1024000
topic.metadata.refresh.interval.ms=30000

We also checked if these properties are being picked.

Based on the documentation, producer.send() queues the message on the async producer's queue.

So, ideally if the queue is full then the enqueue operation should result in an kafka.common.QueueFullException
in 20 ms.

The logs indicate that the enqueue operation is taking more than 20ms (takes around 250ms)
without throwing any exceptions.

Is there any other property that could conflict with queue.enqueue.timeout.ms which is causing
this behavior ?

Or is it possible that the queue is not full & yet the producer.send() call is still taking
around 200ms under peak load ?

Also, could you suggest any other alternatives so that we can either enforce a timeout or
throw an exception in-case the async producer is taking more than a specified amount of time
to queue an event.


Regards,
Dev

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message