kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zor X.L." <xiao.liu...@qq.com>
Subject Re: Consumer groups slow down producer, is this expected?
Date Tue, 19 Sep 2017 01:48:55 GMT
Kafka Version: 0.10.0.1 / 0.10.2.1


在 2017/9/19 9:44, Zor X.L. 写道:
> Hi,
>
>
> Recently in our experiment, we find that even though no resource usage 
> achieve 80%, consumers will slow down producer (which we were not 
> expected), especially when there is no message in the topic.
>
>
> *We wonder if we did something wrong (where?), or this is a Kafka's 
> characteristic(explanation**?lock?).*
>
>
> Here are the details (single partition fan-out):
>
> - Topic: has only 1 partition
>
> - Producer: single thread, max.in.flight.requests.per.connection=1 (we 
> need in-order delivery), batch.size=400000
>
> - Consumers: 0-50 single thread consumers all in different consumer 
> groups
>
> - Broker: 3
>
> - CPU: 2 * E5 2690v4(2.6GHz 14C28T)
>
> - RAM: 384GiB
>
> - HD: 2 * RAID5(6+1, 1.2T), fio direct write throughput > 800MiB/s
>
> - Network: 10GbE
>
>
> Test case:
>
> - replication factor: 3
>
> - producer acks=all
>
> - empty topic
>
> - start producer and consumers at the same time
>
> - Result (number of consumers: producer throughput in MiB/s)
>
>   - 0: 103
>
>   - 1: 88
>
>   - 2: 77
>
>   - 3: 73
>
>   - 4: 67
>
>   - 8: 50
>
>   - 16: 33
>
>   - 30: 20
>
>   - 50: 12
>
> - Resource usage
>
>   - very low CPU Usage (<5%), No single core achieve 100%
>
>   - mid network bandwidth usage (50%-60%)
>
>   - low disk utility
>
>
> We did some additional tests:
>
> - if there are some data in the topic, the producer speed will slow 
> down but not as much. But after consumers catch up, we get the same 
> result
>
> - even if acks=1, the interferance is still there
>
> - tune some system parameters like noatime, but nothing changed.
>
>
>
>



Mime
View raw message