kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Achanta Vamsi Subhash <achanta.va...@flipkart.com>
Subject Batch producer latencies and flush()
Date Tue, 23 Jun 2015 07:33:30 GMT
Hi,

We are using the batch producer of 0.8.2.1 and we are getting very bad
latencies for the topics. We have ~40K partitions now in a 20-node cluster.

- We have many topics and each with messages published to them varying. Ex:
some topics take 10k/sec and other 2000/minute.
- We are seeing latencies of 99th percentile 2sec and 95th percentile of
1sec.
- The current parameters that are tunable are batch size, buffer size and
linger. We monitor the metrics for the new producer and tuned the above
accordingly. Still, we are not able to get any improvements. Batch size in
a sense didn't matter after increasing from 64KB (we increased it till 1MB).
- We also noticed that the record queue time is high (2-3sec).
Documentation describes that this is the time records wait in the
accumulator to be sent.

Later looking at the code in the trunk, I see that the batch size set is
same for all the TopicPartitions and each have their own RecordBatch. Also,
flush() method is added in the latest code.

We want to have an upper bound on the latencies for every message push
irrespective of the incoming rate. Can we achieve it by following logic:

- Wait until X-Kb of batch size / Topic Partition is reached
(or)
- Wait for Y-ms

If either of them is reached, flush the producer records. Can this be part
of the producer code itself? This will avoid the case of records getting
accumulated for 2-3 sec.

Please correct me if the analysis is wrong and suggest me on how do we
improve latencies of the new producer. Thanks.

-- 
Regards
Vamsi Subhash

-- 


------------------------------------------------------------------------------------------------------------------------------------------

This email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the system manager. 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and 
delete this e-mail from your system. If you are not the intended recipient 
you are notified that disclosing, copying, distributing or taking any 
action in reliance on the contents of this information is strictly 
prohibited. Although Flipkart has taken reasonable precautions to ensure no 
viruses are present in this email, the company cannot accept responsibility 
for any loss or damage arising from the use of this email or attachments

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message