kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Binzberger <alexander.binzber...@wingcon.com>
Subject Re: Kafka write throughput tuning
Date Wed, 31 May 2017 08:39:39 GMT
One partition is one file on disk. This is why your throughput per 
partition will not go over 150MB/s.

I can not explain why you do not get 300MB/s for two partition files on 
different disks. It could be related to your hardware. How fast are your 
writes using 2 dd processes?

To get more throughput you would normally use more brokers on more 
machines with a lot of partitions.


Am 17.05.2017 um 11:13 schrieb 陈 建平Chen Jianping:
> Hi Group,
>
> Recently I am trying to turn Kafka write performance to improve throughput. On my Kafka
broker, there are 3 disks (7200 RPM).
> For one disk, the Kafka write throughput can reach 150MB/s. In my opinion, if I send
message to topic test_p3 (which has 3 partitions located on different disk in the same server),
the whole write throughput can reach 450 MB/s due to parallel writing disk. However the test
result is still 150MB/s. Is there any reason that multiple disk doesn’t multiply the write
throughput? And is there any bottleneck for the Kafka write throughput or I need some configuration
to update?
>
> I also try to test sending message to two different topic (whose partition on different
disk of that server), and the total throughput only reach 200 MB/s instead of 300 MB/s as
I expect. Below is my Kafka configuration and setting. Thanks for any idea or advice on it:)
>
> ##Kafka producer setting
> ./kafka-run-class org.apache.kafka.tools.ProducerPerformance --topic test_p3 --num-records
50000000 --record-size 100 --throughput -1 --producer-props acks=0 bootstrap.servers=localhost:9092
buffer.memory=33554432 batch.size=16384
>
> ##OS tuning setting
> net.core.rmem_default = 124928
> net.core.rmem_max = 2048000
> net.core.wmem_default = 124928
> net.core.wmem_max = 2048000
> net.ipv4.tcp_rmem = 4096 87380 4194304
> net.ipv4.tcp_wmem = 4096 87380 4194304
> net.ipv4.tcp_max_tw_buckets = 262144
> net.ipv4.tcp_max_syn_backlog = 1024
> vm.oom_kill_allocating_task = 1
> vm.max_map_count = 200000
> vm.swappiness = 1
> vm.dirty_writeback_centisecs = 500
> vm.dirty_expire_centisecs = 500
> vm.dirty_ratio = 60
> vm.dirty_background_ratio = 5
>
>
> Thanks,
> Eric
>

-- 
Alexander Binzberger
System Designer - WINGcon AG
Tel. +49 7543 966-119

Sitz der Gesellschaft: Langenargen
Registergericht: ULM, HRB 734260
USt-Id.: DE232931635, WEEE-Id.: DE74015979
Vorstand: thomasThomas Ehrle (Vorsitz), Fritz R. Paul (Stellvertreter), Tobias Treß
Aufsichtsrat: Jürgen Maucher (Vorsitz), Andreas Paul (Stellvertreter), Martin Sauter


Mime
View raw message