kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zor X.L." <xiao.liu...@qq.com>
Subject Consumer groups slow down producer, is this expected?
Date Tue, 19 Sep 2017 01:44:40 GMT
Hi,


Recently in our experiment, we find that even though no resource usage 
achieve 80%, consumers will slow down producer (which we were not 
expected), especially when there is no message in the topic.


*We wonder if we did something wrong (where?), or this is a Kafka's 
characteristic(explanation**?lock?).*


Here are the details (single partition fan-out):

- Topic: has only 1 partition

- Producer: single thread, max.in.flight.requests.per.connection=1 (we 
need in-order delivery), batch.size=400000

- Consumers: 0-50 single thread consumers all in different consumer groups

- Broker: 3

- CPU: 2 * E5 2690v4(2.6GHz 14C28T)

- RAM: 384GiB

- HD: 2 * RAID5(6+1, 1.2T), fio direct write throughput > 800MiB/s

- Network: 10GbE


Test case:

- replication factor: 3

- producer acks=all

- empty topic

- start producer and consumers at the same time

- Result (number of consumers: producer throughput in MiB/s)

   - 0: 103

   - 1: 88

   - 2: 77

   - 3: 73

   - 4: 67

   - 8: 50

   - 16: 33

   - 30: 20

   - 50: 12

- Resource usage

   - very low CPU Usage (<5%), No single core achieve 100%

   - mid network bandwidth usage (50%-60%)

   - low disk utility


We did some additional tests:

- if there are some data in the topic, the producer speed will slow down 
but not as much. But after consumers catch up, we get the same result

- even if acks=1, the interferance is still there

- tune some system parameters like noatime, but nothing changed.




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message