kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajiv Kurian <ra...@signalfx.com>
Subject Debugging periodic high kafka log flush times
Date Thu, 17 Sep 2015 23:39:16 GMT
We have a 9 node cluster running 0.8.2.1 that does around 545 thousand
messages(kafka-messages-in) per second. Each of our brokers has 30 G of
memory and 16 cores. We give the brokers themselves 2G of heap. Each broker
ranges from around 33 - 40% cpu utilization. The values for both
kafka-bytes-in and kafka-messages-in are pretty much the same across
servers which seems to suggest that there isn't much imbalance. Our average
message size (kafka-bytes-in / kafka-messages-in) is around 80 bytes for
all brokers.

The 95th percentile of the log flush times on each broker is usually 20 ms
or lower. But periodically (maybe every 4-5 hrs) for a period of about 30
minutes one of the brokers has it's log flush latency 95th go up to 600 ms
or even higher. It is not always the same broker. This causes serious
bubbles in our pipeline and some of the 99th percentile of our producer to
consumer latency goes to 20 seconds from the steady state of under 300 ms.

When the log flush latency goes up the number of under-replicated
partitions also goes up. The CPU on the node does not show any sizable
increase (maybe a couple % absolute if I really squint). The network
traffic on the node goes down, probably from the back-pressure from slow
log flush. It doesn't seem like the affected broker is receiving more
messages than others during the time its log flush time goes up.

It is quite possible that we are under-provisioned since this periodic log
flush bubbles started happening recently as our load started going up. I
don't see anything suspicious in the logs of the machines that are
suffering from the higher log flush times. I want to increase capacity, but
it would be good to understand that this is a provisioning issue and not
something else before I add more machines to the cluster.

Thanks,
Rajiv

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message