kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jrpilat@gmail.com"<jrpi...@gmail.com>
Subject Re: Idle cluster high CPU usage
Date Fri, 22 Sep 2017 23:44:17 GMT
One thing worth trying is hooking up to 1 or more of the brokers via JMX and examining the
running threads;  If that doesn't elucidate the cause, you could move onto sampling or profiling
via JMX to see what's taking up all that CPU.

- Jordan Pilat

On 2017-09-21 07:58, Elliot Crosby-McCullough <elliot.crosby-mccullough@freeagent.com>
wrote: 
> Hello,
> 
> We've been trying to debug an issue with our kafka cluster for several days
> now and we're close to out of options.
> 
> We have 3 kafka brokers associated with 3 zookeeper nodes and 3 registry
> nodes, plus a few streams clients and a ruby producer.
> 
> Two of the three brokers are pinning a core and have been for days, no
> amount of restarting, debugging, or clearing out of data seems to help.
> 
> We've got the logs at DEBUG level which shows a constant flow much like
> this: https://gist.github.com/elliotcm/e66a1ca838558664bab0c91549acb251
> 
> As best as we can tell the brokers are up to date on replication and the
> leaders are well-balanced.  The cluster is receiving no traffic; no
> messages are being sent in and the consumers/streams are shut down.
> 
> From our profiling of the JVM it looks like the CPU is mostly working in
> replication threads and SSL traffic (it's a secured cluster) but that
> shouldn't be treated as gospel.
> 
> Any advice would be greatly appreciated.
> 
> All the best,
> Elliot
> 

Mime
View raw message