kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Craig Pastro <siyo...@gmail.com>
Subject More partitions => less throughput?
Date Fri, 29 Nov 2019 03:10:23 GMT
Hello there,

I was wondering if anyone here could help me with some insight into a
conundrum that I am facing.

Basically, the story is that I am running three Kafka brokers via docker on
a single vm with log.flush.interval.messages = 1 and min.insync.replicas =
2. Then I create two topics: both with replication factor = 3, but one with
one partition and the other with 64.

Then I try to run a benchmark using these topics and what I find is as
follows:

1 partition, 1381.02 records/sec,  685.87 ms average latency
64 partitions, 601.00 records/sec, 1298.18 ms average latency

This is the opposite of what I expected. In neither case am I even close to
the IOPS of what the disk can handle. So what I would like to know is if
there is any obvious reason that I am missing for the slow down with more
partitions?

If it is helpful the docker-compose file and the code to do the
benchmarking can be found at https://github.com/siyopao/kafka-benchmark.
(Any comments or advice on how to make the code better are greatly
appreciated!) The benchmarking code is inspired by and very similar to what
the bin/kafka-producer-perf-test.sh script does.

Thank you!

Best wishes,
Craig

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message