kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajiv Kurian <ra...@signalfx.com>
Subject Debugging high log flush latency on a broker.
Date Tue, 22 Sep 2015 18:46:49 GMT
I have a particular broker(version 0.8.2.1)  in a cluster receiving about
15000 messages/second of around 100 bytes each (bytes-in / messages-in).
This broker has bursts of really high log flush latency p95s. The latency
 sometimes goes to above 1.5 seconds from a steady state of < 20 ms.

Running top on the box  shows bursts of wa (io wait) of over 10% during
these high latency cycles. Running iostat during the same time shows that
it was doing more than 16000 block writes/sec during these periods of high
latency. Running iostat during a normal run shows around 3000 - 7000 block
writes/second. So it does seem like during the latency cycle we are just
trying to do too much IO.

I have tried replacing the broker several times to see if that would help
but it hasn't so far, so my guess is there is something peculiar about one
or more of the partitions that are assigned to that broker. The other
observation is that messages-in/sec and bytes-in/sec DO NOT go up higher
when the broker is misbehaving. I have also noticed that every time I
replace the broker it takes much longer for the new broker to bootstrap
when compared to other brokers that I have replaced. For example it takes
up to 20 minutes before the log flush latency on the newly bootstrapped
broker (with the same ID as the one it replaced) to go to normal levels.

Any hints on what else I can try (replacing broker does not seem to help?
15000 messages/second of around 100 bytes each on a dual SSD, 30 GB RAM and
16 core box would seem very doable. It doesn't bother the other brokers in
the cluster either.

Also any hints on how I can find the exact topic/partitions assigned to
this broker? I know in ZK we can see the partition -> broker mapping, but I
am looking for a broker -> partition mapping. I can't be sure if the load
that is causing this problem is because of leader traffic or follower
traffic. What is weird is that I rarely if ever see other brokers in the
cluster have the same problem. With 3 way replication (leader + 2 replicas)
I'd imagine that the same work load would cause problems on other brokers
too.

Thanks,
Rajiv

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message