kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Petr Novak" <oss.mli...@gmail.com>
Subject Does queue.time still apply for the new Producer?
Date Tue, 02 May 2017 13:38:33 GMT
The documentation reads as:

"As events enter a queue, they are buffered in a queue, until either
queue.time or batch.size is reached. A background thread
(kafka.producer.async.ProducerSendThread) dequeues the batch of data and
lets the kafka.producer.EventHandler serialize and send the data to the
appropriate kafka broker partition."


It clearly seems yes. But isn't it only for the old producer? Because
queue.time is not part of Producer configuration options as defined in the
same documentation. I can't find it in ProducerConfig source
/kafka/clients/producer/ProducerConfig.java> . Looking into source code
there is deprecated ProducerSendThread
66/core/src/main/scala/kafka/producer/async/ProducerSendThread.scala>  which
uses queueTime. But at first glance I can't see where it is used in the new
66/core/src/main/scala/kafka/producer/async/DefaultEventHandler.scala>  or
am I looking at the wrong place? The Producer
er/Producer.scala>  asyncSend seems to use only queueEnqueueTimeoutMs.

I'm trying to figure out if sending can be blocked by queue.time or
batch.size in the new producer. I thought that it is not and that sending
loop dequeues as fast as possible, creates batches of max batch.size from
what is currently there. And how fast it is dequed is limited only by linger
and max.in.flight.requests.per.connection. Or it still waits for either
queue.time to batch.size to fill? I got confused by the documentation which
suggests otherwise.

I can't wrap my head around how linger would help if Producer waits for
queue.time or batch.size. What can I control with linger I can't control
with the combination of other two in this case. And how they all play
together? (If all still applies)

Many thanks for any clarification,

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message