kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hans Jespersen <h...@confluent.io>
Subject Re: How to increase network throughput of Kafka cluster?
Date Sat, 01 Apr 2017 20:51:26 GMT
What replication factor are you using? If you have a default replication factor = 3 then a
publish rate of 1.4 Gbps is actually 1.4 Gbps *3 = 4.2 Gbps of network traffic. If you are
also consuming at the same time then it’s actually 4.2 Gbps + 1.4 Gbps = 5.6 Gbps.  

You would completely saturate the network if you added a second producer and consumer at those
rates (if your storage system can keep up to the network bandwidth).

-hans



> On Apr 1, 2017, at 10:25 AM, Archie <anubhavnidhi1009@gmail.com> wrote:
> 
> I have set up my kafka cluster in a network with 9.3 Gbps links. And am
> using the bin/kafka-producer-perf-test.sh script to test the throughput
> performance of kafka for a topic which has 1 partition.
> 
> Right now I am only able to use 1.4 Gbps of network link, i.e. the script
> returns a throughput of 1.4 Gbps (179 MBps)
> 
> My basic question is how can I increase the throughput of kafka to
> completely saturate the network?
> 
> Thanks,
> Archie


Mime
View raw message