kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: benchmark kafka on 10GbE network
Date Wed, 19 Nov 2014 04:34:17 GMT
Yeah this will involve some experimentation.

The metrics are visible with jconsole or another jmx viewer.

It may also be worth looking at the cpu usage per-thread (e.g. start top
and press 't' I think).

Another simple test for broker vs client as the bottleneck is just to start
another producer or consumer and see if that improves throughput (if so it
is probably a client bottleneck).

-Jay

On Tue, Nov 18, 2014 at 4:44 PM, Manu Zhang <owenzhang1990@gmail.com> wrote:

> Thanks Jay for the quick response.
>
> Yes, it's a single producer and consumer both configured with multiple
> threads but I'm not using the new producer.
> CPU is typically 50% utilized on client and merely used on broker. Disks
> aren't busy either as a lot of data are cached in memory.
> Would you please give a link for the producer metrics you are referring to
> ?
>
> Thanks,
> Manu
>
> On Wed, Nov 19, 2014 at 2:39 AM, Jay Kreps <jay.kreps@gmail.com> wrote:
>
> > Hey Manu,
> >
> > I'm not aware of a benchmark on 10GbE. I'd love to see that though.
> Diving
> > into the results may help us find bottlenecks hidden by the slower
> network.
> >
> > Can you figure out where the bottleneck is in your test? I assume this
> is a
> > single producer and consumer instance and you are using the new producer
> as
> > in those benchmarks?
> >
> > This can be slightly tricky as it can be cpu or I/O on either the clients
> > or the brokers. You basically have to look at top, iostat, and the jmx
> > metrics for clues. The producer has good metrics that explain whether it
> is
> > spending most of its time waiting or sending data. Not sure if there is a
> > similar diagnostic for the consumer.
> >
> > -Jay
> >
> > On Tue, Nov 18, 2014 at 5:10 AM, Manu Zhang <owenzhang1990@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I have been trying out kafka benchmarks described in Jay's
> > > benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
> > > <
> > >
> >
> https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machine
> > > >.
> > > I'm able to get similar results on a 4-node GbE network whose in-bytes
> > > could be saturated at 120MB/s. However, on a 4-node, 10GbE network, I
> can
> > > not get in-bytes higher than 150MB/s. *Has anyone benchmarked kafka on
> a
> > > 10GbE network ? Any rule of thumb on 10GbE network for configurations
> of
> > > broker, producer and consumer ? *
> > >
> > > My kafka version is 0.8.1.1 and I've created a topic with 8 partitions
> > with
> > > 1 replica distributed evenly among the 4 nodes. Message size is 100
> > bytes.
> > > I use all the default kafka settings.
> > > My cluster has 4 nodes, where each node has 32 cores, 128MB RAM and 3
> > disks
> > > for kafka.
> > >
> > > I've tried increasing message size to 1000 bytes which improved
> > producer's
> > > throughput but not consumer's.
> > >
> > >
> > > Thanks,
> > > Manu
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message