kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Owhadi <eric.owh...@esgyn.com>
Subject RE: More partitions => less throughput?
Date Sat, 30 Nov 2019 17:53:09 GMT
What is happening imho is that when you have multiple partitions, each consumer will fetch
data from its partition and find only 1/64th the amount of data (compared to the single partition
case) to send every time it is its turn to send stuff. Therefore you end up having a more
chatty situation, where each push to broker carry too small number of messages, compared to
the single partition case that optimize can perform the same function but each set of message
send to broker contains higher message count.
Eric

-----Original Message-----
From: Craig Pastro <siyopao@gmail.com> 
Sent: Thursday, November 28, 2019 9:10 PM
To: users@kafka.apache.org
Subject: More partitions => less throughput?

External

Hello there,

I was wondering if anyone here could help me with some insight into a conundrum that I am
facing.

Basically, the story is that I am running three Kafka brokers via docker on a single vm with
log.flush.interval.messages = 1 and min.insync.replicas = 2. Then I create two topics: both
with replication factor = 3, but one with one partition and the other with 64.

Then I try to run a benchmark using these topics and what I find is as
follows:

1 partition, 1381.02 records/sec,  685.87 ms average latency
64 partitions, 601.00 records/sec, 1298.18 ms average latency

This is the opposite of what I expected. In neither case am I even close to the IOPS of what
the disk can handle. So what I would like to know is if there is any obvious reason that I
am missing for the slow down with more partitions?

If it is helpful the docker-compose file and the code to do the benchmarking can be found
at https://github.com/siyopao/kafka-benchmark.
(Any comments or advice on how to make the code better are greatly
appreciated!) The benchmarking code is inspired by and very similar to what the bin/kafka-producer-perf-test.sh
script does.

Thank you!

Best wishes,
Craig
Mime
View raw message