kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niklas Ström <str...@gmail.com>
Subject Expected client producer/consumer CPU utilization when idle
Date Thu, 01 Dec 2016 14:20:34 GMT
Hello all

Can anyone say something about what CPU utilization we can expect for a
producer/consumer process that is idle, i.e. not producing or consuming any
messages? Should it be like 0%? What is your experience?

We have a small program with a few kafka producers and consumers and we are
concerned about the CPU utilization when it is idle. Currently it uses on
average 6% CPU when idle and about 12% when under a low load of 4 messages
per second. In a scaled down test program with only one producer and one
consumer the corresponding figures are 2.6% and 4.8%.

We want to have a lot of different small processes each producing and
consuming a number of topics so we really want to minimize the CPU
utilization of each process, they will not all do heavy work at the same
time so occasional high loads for a process is not a problem, but if all
processes are using the CPU when not really doing anything useful we might
get into problem.

Tests are run with one local kafka broker, on a machine with Intel i7 2
Cores @ 3Ghz, 16 GB RAM, in a virtual environment with Ubuntu 14.04.4 LTS.
Using java kafka client 0.10.0.0. Our only kafka configuration parameter
that is not default is KAFKA_FETCH_MIN_BYTES that is 1 in order to reduce
latency.

So far we have not run so many processes at the same time, but we fear that
if we try to run a couple of hundred of these processes we will get into
problem.

Would greatly appreciate any input. If not else, please tell me the CPU
utilization of your producer/consumer processes when they are not really
under load. Just so I can conclude if our program behaves as expected or if
we have any configuration or environment issues

Thanks
Niklas Ström

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message