kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niklas Ström <str...@gmail.com>
Subject Configuration for low latency and low cpu utilization? java/librdkafka
Date Thu, 08 Dec 2016 15:44:13 GMT
Use case scenario:
We want to have a fairly low latency, say below 20 ms, and we want to be
able to run a few hundred processes (on one machine) both producing and
consuming a handful of topics. The throughput is not high, lets say on
average 10 messages per second for each process. Most messages are 50-500
bytes large, some may be a few kbytes.

How should we adjust the configuration parameters for our use case?

Our experiments so far gives us a good latency but at the expence of CPU
utilization. Even with a bad latency, the CPU utilization is not
satisfying. Since we will have a lot of processes we are concerned that
short poll loops will cause an overconsumption of CPU capacity. We are
hoping we might have missed some configuration parameter or that we have
some issues with our environment that we can find and solve.

We are using both the java client and librdkafka and see similar CPU issues
in both clients.

We have looked at recommendations from:
https://github.com/edenhill/librdkafka/wiki/How-to-decrease-message-latency
The only thing that seems to really make a difference for librdkafka is
socket.blocking.max.ms, but reducing that also makes the CPU go up.

I would really appreciate input on configuration parameters and of any
experience with environment issues that has caused CPU load. Or is our
scenario not feasible at all?

Cheers

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message