kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Issues with Kafka async producer's enqueue timeout
Date Thu, 20 Nov 2014 23:38:12 GMT
queue.enqueue.timeout.ms only applies if the producer is configured in
async mode.

Thanks,

Jun

On Thu, Nov 20, 2014 at 5:01 AM, Devendra Tagare <
devendra.tagare@pubmatic.com> wrote:

> Hi,
>
> We are using an async producer to send data to a kafka cluster.The event
> rate at peak is around 250 events/second of size 25KB each.
>
> In the producer code base we have added specific debug statements to
> capture the time taken to create a producer,create a keyed message with a
> byte payload & send the message.
>
> We have added the below properties to the ProducerConfig
>
> queue.enqueue.timeout.ms=20
> send.buffer.bytes=1024000
> topic.metadata.refresh.interval.ms=30000
>
> We also checked if these properties are being picked.
>
> Based on the documentation, producer.send() queues the message on the
> async producer's queue.
>
> So, ideally if the queue is full then the enqueue operation should result
> in an kafka.common.QueueFullException in 20 ms.
>
> The logs indicate that the enqueue operation is taking more than 20ms
> (takes around 250ms) without throwing any exceptions.
>
> Is there any other property that could conflict with
> queue.enqueue.timeout.ms which is causing this behavior ?
>
> Or is it possible that the queue is not full & yet the producer.send()
> call is still taking around 200ms under peak load ?
>
> Also, could you suggest any other alternatives so that we can either
> enforce a timeout or throw an exception in-case the async producer is
> taking more than a specified amount of time to queue an event.
>
>
> Regards,
> Dev
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message