kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 陈 建平Chen Jianping <chenjianp...@agora.io>
Subject Kafka write throughput tuning
Date Wed, 17 May 2017 09:13:41 GMT
Hi Group,

Recently I am trying to turn Kafka write performance to improve throughput. On my Kafka broker,
there are 3 disks (7200 RPM).
For one disk, the Kafka write throughput can reach 150MB/s. In my opinion, if I send message
to topic test_p3 (which has 3 partitions located on different disk in the same server), the
whole write throughput can reach 450 MB/s due to parallel writing disk. However the test result
is still 150MB/s. Is there any reason that multiple disk doesn’t multiply the write throughput?
And is there any bottleneck for the Kafka write throughput or I need some configuration to
update?

I also try to test sending message to two different topic (whose partition on different disk
of that server), and the total throughput only reach 200 MB/s instead of 300 MB/s as I expect.
Below is my Kafka configuration and setting. Thanks for any idea or advice on it:)

##Kafka producer setting
./kafka-run-class org.apache.kafka.tools.ProducerPerformance --topic test_p3 --num-records
50000000 --record-size 100 --throughput -1 --producer-props acks=0 bootstrap.servers=localhost:9092
buffer.memory=33554432 batch.size=16384

##OS tuning setting
net.core.rmem_default = 124928
net.core.rmem_max = 2048000
net.core.wmem_default = 124928
net.core.wmem_max = 2048000
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 87380 4194304
net.ipv4.tcp_max_tw_buckets = 262144
net.ipv4.tcp_max_syn_backlog = 1024
vm.oom_kill_allocating_task = 1
vm.max_map_count = 200000
vm.swappiness = 1
vm.dirty_writeback_centisecs = 500
vm.dirty_expire_centisecs = 500
vm.dirty_ratio = 60
vm.dirty_background_ratio = 5


Thanks,
Eric

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message