kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Stelly <cdste...@gmail.com>
Subject Re: Tuning for high RAM and 10GBe
Date Sun, 09 Oct 2016 17:19:50 GMT
Hey Eno, thanks

Yes, localhost would mean testing SSD, maybe moving to a remote producer in
the future over 10GBe but baby steps first.

My record size is pretty flexible - I've tested from 4k to 16M and the
results are +- 10MB/s.

SSD is a samsung 850, claiming 100k IOPS and >500MB/s read/write.

Again, thanks!

On Sun, Oct 9, 2016 at 12:55 PM, Eno Thereska <eno.thereska@gmail.com>
wrote:

> Hi Chris,
>
> A couple of things: looks like you're primarily testing the SSD if you're
> running on the localhost, right? The 10GBe shouldn't matter in this case.
>
> The performance will depend a lot on the record sizes you are using. To
> get a ballpark number if would help to know the SSD type, would you be
> willing to share it? We can take it from there.
>
> Thanks
> Eno
>
> > On 9 Oct 2016, at 17:28, Christopher Stelly <cdstelly@gmail.com> wrote:
> >
> > Hello,
> >
> > The last thread available regarding 10GBe is about 2 years old, with no
> > obvious recommendations on tuning.
> >
> > Is there a more complex tuning guide than the example production config
> > available on Kafka's main site? Anything other than the list of possible
> > configs?
> >
> > I currently have access to a rather substantial academic cluster to test
> > on, including multiple machines with the following hardware:
> >
> > 10GBe NICs
> > 250GB RAM each
> > SSDs on each
> > (also, optional access to single NVMe)
> >
> > Using Golang's Kafka interface, I can only seem to get about 80MB/s on
> the
> > producer pushing to logs on the localhost, using no replication and
> reading
> > from/logging to SSD. If it helps, I can post my configs. I've tried
> > fiddling with a bunch of broker configs as well as producer configs,
> > raising the memory limits, max message size, io&network threads etc.
> >
> > Since the last post from 2014 indicates that there is no public
> > benchmarking for 10GBe, I'd be happy to run benchmarks /publish results
> on
> > this hardware if we can get it tuned up properly.
> >
> > What kind of broker/producer/consumer settings would you recommend?
> >
> > Thanks!
> > - chris
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message