kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eno Thereska <eno.there...@gmail.com>
Subject Re: Tuning for high RAM and 10GBe
Date Sun, 09 Oct 2016 16:55:20 GMT
Hi Chris,

A couple of things: looks like you're primarily testing the SSD if you're running on the localhost,
right? The 10GBe shouldn't matter in this case.

The performance will depend a lot on the record sizes you are using. To get a ballpark number
if would help to know the SSD type, would you be willing to share it? We can take it from
there.

Thanks
Eno

> On 9 Oct 2016, at 17:28, Christopher Stelly <cdstelly@gmail.com> wrote:
> 
> Hello,
> 
> The last thread available regarding 10GBe is about 2 years old, with no
> obvious recommendations on tuning.
> 
> Is there a more complex tuning guide than the example production config
> available on Kafka's main site? Anything other than the list of possible
> configs?
> 
> I currently have access to a rather substantial academic cluster to test
> on, including multiple machines with the following hardware:
> 
> 10GBe NICs
> 250GB RAM each
> SSDs on each
> (also, optional access to single NVMe)
> 
> Using Golang's Kafka interface, I can only seem to get about 80MB/s on the
> producer pushing to logs on the localhost, using no replication and reading
> from/logging to SSD. If it helps, I can post my configs. I've tried
> fiddling with a bunch of broker configs as well as producer configs,
> raising the memory limits, max message size, io&network threads etc.
> 
> Since the last post from 2014 indicates that there is no public
> benchmarking for 10GBe, I'd be happy to run benchmarks /publish results on
> this hardware if we can get it tuned up properly.
> 
> What kind of broker/producer/consumer settings would you recommend?
> 
> Thanks!
> - chris


Mime
View raw message