kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dominik Werder <dominik.wer...@gmail.com>
Subject Kafka write MB/s
Date Mon, 03 Jul 2017 08:47:44 GMT
Hello everyone,

I'm interested in the maximum transfer rate of a Kafka broker. There
is a fair bit of performance figures in terms of messages per second,
but I'm more interested in MB/s with rather large messages, let's say
500 kB as an example.

I tried testing by writing with the apache kafka performance test
tool, and with librdkafka.

The hardware can write continously 700 MB/s to local storage, but the
maximum I can achieve with Kafka is about 260 MB/s with acks=0.

I varied the various server side and client side options, batch sizes,
message sizes, send and receive buffer sizes, network buffer sizes,
JVM heap size. I tested both over loopback and over 10Gb ethernet,
same results.

Before digging into this much more, I wondered whether somebody has
experience with writing to a single Kafka broker at these transfer
rates?

Best regards
Dominik

Mime
View raw message