kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "BOOPATHY, Jegan" <jegan.boopa...@airbus.com>
Subject kafka perf test 5MB message
Date Fri, 20 Nov 2015 09:21:29 GMT
Hello,

I am using Kafka for my thesis project. I need to do some performance tests. I managed to
send 5 MB data in producer performace test like below. I have 3 Zookeeper, 2 brokers, 1 producer
and 1 consumer in a single machine for testing. I tried with consumer perf test. But it failed.
Kindly help me out with the consumer perf test with 5MB data.

bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance perftest 1000 5242880
-1 acks=1 bootstrap.servers=localhost:9093 max.request.size=2147483646
184 records sent, 36.6 records/sec (182.90 MB/sec), 172.8 ms avg latency, 457.0 max latency.
102 records sent, 20.4 records/sec (101.82 MB/sec), 337.2 ms avg latency, 2276.0 max latency.
157 records sent, 31.2 records/sec (155.85 MB/sec), 218.1 ms avg latency, 756.0 max latency.
109 records sent, 20.2 records/sec (101.17 MB/sec), 307.5 ms avg latency, 1796.0 max latency.
167 records sent, 33.3 records/sec (166.73 MB/sec), 218.9 ms avg latency, 1794.0 max latency.
137 records sent, 27.1 records/sec (135.43 MB/sec), 241.8 ms avg latency, 1171.0 max latency.
1000 records sent, 28.738110 records/sec (143.69 MB/sec), 234.60 ms avg latency, 2276.00 ms
max latency, 167 ms 50th, 496 ms 95th, 1419 ms 99th, 2276 ms 99.9th.

bin/kafka-consumer-perf-test.sh --zookeeper localhost:2181 --messages 1000 --topic perftest
--threads 10 --fetch-size 5242880 --socket-buffer-size 2147483646
start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
[2015-11-20 10:19:39,096] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-5
for topic perftest (kafka.consumer.RangeAssignor)
[2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-3
for topic perftest (kafka.consumer.RangeAssignor)
[2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-7
for topic perftest (kafka.consumer.RangeAssignor)
[2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-2
for topic perftest (kafka.consumer.RangeAssignor)
[2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-6
for topic perftest (kafka.consumer.RangeAssignor)
[2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-4
for topic perftest (kafka.consumer.RangeAssignor)
[2015-11-20 10:19:39,099] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-8
for topic perftest (kafka.consumer.RangeAssignor)
[2015-11-20 10:19:39,099] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-9
for topic perftest (kafka.consumer.RangeAssignor)
2015-11-20 10:19:40:346, 2015-11-20 10:19:45:382, 5242880, 8.0000, 222.2222, 2, 55.5556

This is how I created a topic with two partitions. Actually, how partitions help in the performance?
What does it actually do?

bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic perftest --replication-factor
2 --partitions 2


Regards
Jegan

<html><head></head><body><font color="black" face="arial" size="2">
The information in this e-mail is confidential. The contents may not be disclosed or used
by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised.
If you are not the intended recipient, please notify Airbus immediately and delete this e-mail.
Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as
it has been sent over public networks. If you have any concerns over the content of this message
or its Accuracy or Integrity, please contact Airbus immediately.
All outgoing e-mails from Airbus are checked using regularly updated virus scanning software
but you should take whatever measures you deem to be appropriate to ensure that this message
and any attachments are virus free.
</font>
</body>
</html>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message