kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: Expected client producer/consumer CPU utilization when idle
Date Mon, 05 Dec 2016 06:11:50 GMT
If completely idle, producers shouldn't need to do anything beyond very
infrequent metadata updates (once ever few minutes). Consumers, however,
will have some ongoing work -- they will always issues fetch requests (to
get more data) and heartbeats (to indicate they are still alive). But these
shouldn't create a ton of overhead since each only needs a small amount of
work every few seconds if there's no data flowing.

Note that the setting you mention as not default (fetch.min.bytes) is, in
fact, the default value of 1.

If you're concerned about the CPU usage, I'd suggest running your processes
under a profiler for a bit to determine where the CPU time is going.


On Thu, Dec 1, 2016 at 6:20 AM, Niklas Ström <stromn@gmail.com> wrote:

> Hello all
> Can anyone say something about what CPU utilization we can expect for a
> producer/consumer process that is idle, i.e. not producing or consuming any
> messages? Should it be like 0%? What is your experience?
> We have a small program with a few kafka producers and consumers and we are
> concerned about the CPU utilization when it is idle. Currently it uses on
> average 6% CPU when idle and about 12% when under a low load of 4 messages
> per second. In a scaled down test program with only one producer and one
> consumer the corresponding figures are 2.6% and 4.8%.
> We want to have a lot of different small processes each producing and
> consuming a number of topics so we really want to minimize the CPU
> utilization of each process, they will not all do heavy work at the same
> time so occasional high loads for a process is not a problem, but if all
> processes are using the CPU when not really doing anything useful we might
> get into problem.
> Tests are run with one local kafka broker, on a machine with Intel i7 2
> Cores @ 3Ghz, 16 GB RAM, in a virtual environment with Ubuntu 14.04.4 LTS.
> Using java kafka client Our only kafka configuration parameter
> that is not default is KAFKA_FETCH_MIN_BYTES that is 1 in order to reduce
> latency.
> So far we have not run so many processes at the same time, but we fear that
> if we try to run a couple of hundred of these processes we will get into
> problem.
> Would greatly appreciate any input. If not else, please tell me the CPU
> utilization of your producer/consumer processes when they are not really
> under load. Just so I can conclude if our program behaves as expected or if
> we have any configuration or environment issues
> Thanks
> Niklas Ström


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message