kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xinyi Su <xiny...@gmail.com>
Subject Kafka long tail latency issue
Date Tue, 03 Feb 2015 09:37:56 GMT
Hi,
I am building Kafka cluster and run producer perf test to get Kafka latency
performance.
>From test result, I notice that the long tail latency is very high and
increased with time passing by although the 99.9% result looks very good.
The worst latency can reach more than 1 second. Besides, disk utilization
is always very low, never more than 1%. I also try to tune
log.flush.interval.ms from 1000ms to 200ms. It does not help much.

Below is the max latency chart, Y axis represents the max latency in
millisecond, X axis represents the time elapsed in milliseconds. From
chart, we can see the latency increasing from about 10ms to 1095ms
gradually.

[image: Inline image]

Kafka cluster is built up with 4 hosts. The version is 2.9.2-0.8.2-beta.
The PerfTopic15 topic is created with 3 partition and 3 replication.

Here is my perf script usage:
-bash-4.1$ bin/kafka-producer-perf-test.sh   --broker-list <broker
list> --topics *PerfTopic15* --sync --initial-message-id 1 --messages
200000 --csv-reporter-enabled --metrics-dir /tmp/PerfTopic15_1
--message-send-gap-ms 20* --request-num-acks -1* --batch-size 1

-bash-4.1$ bin/kafka-topics.sh  --zookeeper <zkHost>:2181  --describe
--topic *PerfTopic15*
Topic:PerfTopic15 PartitionCount:3 ReplicationFactor:3 Configs:
Topic: PerfTopic15 Partition: 0 Leader: 3 Replicas: 3,4,1 Isr: 3,4,1
Topic: PerfTopic15 Partition: 1 Leader: 4 Replicas: 4,1,2 Isr: 4,1,2
Topic: PerfTopic15 Partition: 2 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3

I expect the worst latency not exceed 100 milliseconds. But the test result
is very discouraging. Do you have some points about Kafka long tail latency
issue?

Hope for your reply! Thanks in advance!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message