kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elliot Crosby-McCullough <elliot.crosby-mccullo...@freeagent.com>
Subject Re: Idle cluster high CPU usage
Date Thu, 21 Sep 2017 14:02:45 GMT
551 partitions, broker configs are:
https://gist.github.com/elliotcm/3a35f66377c2ef4020d76508f49f106b

We tweaked it a bit from standard recently but that was as part of the
debugging process.

After some more experimentation I'm seeing the same behaviour at about half
the CPU after creating one 50 partition topic in an otherwise empty cluster.

On 21 September 2017 at 14:20, Ismael Juma <ismael@juma.me.uk> wrote:

> A couple of questions: how many partitions in the cluster and what are your
> broker configs?
>
> On Thu, Sep 21, 2017 at 1:58 PM, Elliot Crosby-McCullough <
> elliot.crosby-mccullough@freeagent.com> wrote:
>
> > Hello,
> >
> > We've been trying to debug an issue with our kafka cluster for several
> days
> > now and we're close to out of options.
> >
> > We have 3 kafka brokers associated with 3 zookeeper nodes and 3 registry
> > nodes, plus a few streams clients and a ruby producer.
> >
> > Two of the three brokers are pinning a core and have been for days, no
> > amount of restarting, debugging, or clearing out of data seems to help.
> >
> > We've got the logs at DEBUG level which shows a constant flow much like
> > this: https://gist.github.com/elliotcm/e66a1ca838558664bab0c91549acb251
> >
> > As best as we can tell the brokers are up to date on replication and the
> > leaders are well-balanced.  The cluster is receiving no traffic; no
> > messages are being sent in and the consumers/streams are shut down.
> >
> > From our profiling of the JVM it looks like the CPU is mostly working in
> > replication threads and SSL traffic (it's a secured cluster) but that
> > shouldn't be treated as gospel.
> >
> > Any advice would be greatly appreciated.
> >
> > All the best,
> > Elliot
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message