kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Stelly <cdste...@gmail.com>
Subject Tuning for high RAM and 10GBe
Date Sun, 09 Oct 2016 16:28:17 GMT
Hello,

The last thread available regarding 10GBe is about 2 years old, with no
obvious recommendations on tuning.

Is there a more complex tuning guide than the example production config
available on Kafka's main site? Anything other than the list of possible
configs?

I currently have access to a rather substantial academic cluster to test
on, including multiple machines with the following hardware:

10GBe NICs
250GB RAM each
SSDs on each
(also, optional access to single NVMe)

Using Golang's Kafka interface, I can only seem to get about 80MB/s on the
producer pushing to logs on the localhost, using no replication and reading
from/logging to SSD. If it helps, I can post my configs. I've tried
fiddling with a bunch of broker configs as well as producer configs,
raising the memory limits, max message size, io&network threads etc.

Since the last post from 2014 indicates that there is no public
benchmarking for 10GBe, I'd be happy to run benchmarks /publish results on
this hardware if we can get it tuned up properly.

What kind of broker/producer/consumer settings would you recommend?

Thanks!
- chris

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message