kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Håkon Åmdal <hakon.amd...@gmail.com>
Subject Re: performance test using real data - comparing throughput & latency
Date Sat, 16 Sep 2017 06:58:20 GMT
Revin,

We instrument our data pipeline and Kafka applications using both with wall
time within our consumers and stream processors, and in a separate
application that measures end-to-end latency in our data processing
pipelines. We report these metrics to a metric aggregator, which in our
case is Datadog. Using these metrics, we perform isolated performance
experiments on production volume data.

I would be very happy to hear about your results as well.

Best,
Håkon

lør. 16. sep. 2017 kl. 02:13 skrev Matt Andruff <matt.andruff@gmail.com>:

> Look,. I'm a huge fan of sending identical data and using plane old 'wall
> time' and averaging a couple runs to make sure you remove any whoops.
>
> You can use fancy tools for reporting but in the real world wall time still
> is the most critical factor.  And let's face it it's also simple to
> measure.
>
> I personally would love to hear about your results.  If you'd be willing to
> share.
>
> On Fri, Sep 15, 2017, 15:01 Revin Chalil <rchalil@expedia.com> wrote:
>
> > Any thoughts on the below will be appreciated. Thanks.
> >
> >
> > On 9/13/17, 5:00 PM, "Revin Chalil" <rchalil@expedia.com> wrote:
> >
> >     We are testing kafka’s performance with the real prod data and plan
> to
> > test things like the below. We would have producers publishing and
> > consumers processing production data on a separate non-prod kafka
> cluster.
> >
> >
> >       *   Impact of number of Partitions per Topic on throughput and
> > latency on Producer & Consumer
> >       *   Impact of scaling-up Brokers on throughput and latency
> >       *   adding more brokers Vs adding more Disk on existing Brokers.
> How
> > does the network interface usage differ?
> >       *   cost of Replication on Throughput and Latency
> >       *   impact of Broker vm.swappiness = 60 Vs vm.swappiness = 1
> >       *   partitions on a Broker pointing to single Disk Vs multiple
> Disks
> >       *   EXT4 Vs XFS Filesystem on broker
> >       *   behavior when Broker “num.io<http://num.io/>.threads” is
> > increased from 8 to higher value
> >       *   behavior when Broker “num.network.threads” is increased from 3
> > to higher value
> >       *   behavior when the data segment size is increased from 1 GB
> > (current setting)
> >       *   producer “acks = 1” Vs “acks = all” (current setting) impact on
> > throughput and latency
> >       *   producer sending with Compression enabled (snappy?) Vs sending
> > without Compression
> >       *   setting producer batch-size (memory based) Vs record-count
> > (current setting) per batch sent to Kafka
> >       *   impact of message size throughput
> >       *   Consumers fetching records from page-cache Vs fetching records
> > from Disk
> >
> >
> >     Ideally, the metrics we would like to compare for each test are
> > (please let know if there are anything else to be compared)
> >
> >       *   Producer write Throughput
> >       *   Producer write Latency (ms)
> >       *   Consumption Throughput
> >       *   Consumption Latency (ms)
> >       *   End-to-end Latency
> >
> >     What would be the right tools to collect and compare the above
> metrics
> > against different Tests? I have setup kafka-monitor but couldn’t find how
> > to track the throughput and latency. Kafka-web-console seems to have some
> > of these available? Kafka-Manager? Burrow? Anything else? Thank you.
> >
> >     Since we are going to use our own producers and consumers, I do not
> > think it makes sense to use tools like kafka-consumer-perf-test.sh or
> > kafka-producer-perf-test.sh, but please correct if I am wrong.
> >
> >     Thanks,
> >     Revin
> >
> >
> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message