kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Brown <tombrow...@gmail.com>
Subject Re: More partitions => less throughput?
Date Sat, 30 Nov 2019 16:57:10 GMT
I think the number of partitions needs to be tuned to the size of the
cluster; 64 partitions on what is essentially a single box seems high. Do
you know what hardware you will be deploying on in production? Can you run
your benchmark on that instead of a vm?

—Tom

On Thursday, November 28, 2019, Craig Pastro <siyopao@gmail.com> wrote:

> Hello there,
>
> I was wondering if anyone here could help me with some insight into a
> conundrum that I am facing.
>
> Basically, the story is that I am running three Kafka brokers via docker on
> a single vm with log.flush.interval.messages = 1 and min.insync.replicas =
> 2. Then I create two topics: both with replication factor = 3, but one with
> one partition and the other with 64.
>
> Then I try to run a benchmark using these topics and what I find is as
> follows:
>
> 1 partition, 1381.02 records/sec,  685.87 ms average latency
> 64 partitions, 601.00 records/sec, 1298.18 ms average latency
>
> This is the opposite of what I expected. In neither case am I even close to
> the IOPS of what the disk can handle. So what I would like to know is if
> there is any obvious reason that I am missing for the slow down with more
> partitions?
>
> If it is helpful the docker-compose file and the code to do the
> benchmarking can be found at https://github.com/siyopao/kafka-benchmark.
> (Any comments or advice on how to make the code better are greatly
> appreciated!) The benchmarking code is inspired by and very similar to what
> the bin/kafka-producer-perf-test.sh script does.
>
> Thank you!
>
> Best wishes,
> Craig
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message