kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Péter Sinóros-Szabó <peter.sinoros-sz...@transferwise.com.INVALID>
Subject Kafka broker using too much CPU?
Date Wed, 04 Mar 2020 10:23:35 GMT
Hi,

I read here and there that Kafka is not CPU intensive, but mostly disk and
network. Seems to be reasonable, but that's not what I see on my monitoring.

Could anyone help me to see if the CPU usage I see is about the expected
usage or there is something how we use Kafka that makes it more CPU hungry
that it should be?

Running 6 Kafka brokers v1.1.1 each on AWS EC2 m5 instances with 32GB ram,
8 CPUs
We have 830 topics that in sum has 3050 partitions with varying usage
patterns.
Inbound traffic: 3-8MB/sec, 8-15k msgs/sec
Outbound traffic: 10-50MB/sec
I see a total of 21k client connections (TCP connection to broker port), so
guess that is 21k Kafka clients connecting to the cluster.

So CPU usage is at minimum 2-3.5 CPUs on baseline traffic, 4-5 CPUs with
occasional spikes to 7 CPUs.

I feel that this CPU usage is more than what is expected, but I may
be wrong.

What do you think?

Thanks,
Peter

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message