kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bert Corderman <bertc...@gmail.com>
Subject performance testing data to share
Date Fri, 25 Apr 2014 22:11:39 GMT
I have been testing kafka for the past week or so and figured I would share
my results so far.


I am not sure if the formatting will keep in email but here are the results
in a google doc...all 1,100 of them


https://docs.google.com/spreadsheets/d/1UL-o2MiV0gHZtL4jFWNyqRTQl41LFdM0upjRIwCWNgQ/edit?usp=sharing



One thing I found is there appears to be a bottleneck in
kafka-producer-perf-test.sh


The servers I used for testing have 12 7.2K drives and 16 cores.  I was NOT
unable to scale the broker past 350MBsec when adding drives even though I
was able to get 150MBsec from a single drive.  I wanted to determine the
source of the low utilization.


I tired changing the following

·        log.flush.interval.messages on the broker

·        log.flush.interval.ms flush on the broker

·        num.io.threads on the broker

·        thread settings on the producer

·        producer message  sizes

·        producer batch sizes

·        different number of topics (which impact the number of drives)

None of the above had any impact.  The last thing I tried was running
multiple producers which had a very noticeable impact.  As previously
mentioned I had already tested the thread setting of the producer and found
it to scale when increasing the thread count from 1,2,4 and 8.  After that
it plateaued so I had been using 8 threads for each test.   To show the
impact on number of producers I created 12 topics with partition counts
from 1 to 12.    I used a single broker with no replication and configured
the producer(s) to send 10 million 2200 byte messages in batches of 400
with no ack.


Running with three producers has almost double the throughput that one
producer will have.


Other Key points learned so far

·        Ensure you are using correct network interface.  ( use
advertised.host.name if the servers have multiple interfaces)

·        Use batching on the producer – With a single broker sending 2200
byte messages in batches of 200 resulted in  283MBsec vs. a batch size of 1
was 44MBsec

·        The message size, the configuration of request.required.acks and
the number of replicas (only when ack is set to all) had the most influence
on the overall throughput.

·        The following table shows results of testing with messages sizes
of 200, 300, 1000 and 2200 bytes on a three node cluster.  Each message
size was tested with the three available ack modes (NONE, LEADER and ALL)
and with replication of two and three copies.   Having three copies of data
is recommended, however both are included for reference.

*Replica=2*

*Replica=3*

*message.size*

*acks*

*MB.sec*

*nMsg.sec*

*MB.sec*

*nMsg.sec*

*Per Server MB.sec*

*Per Server nMsg.sec*

200

NONE

251

1,313,888

237

1,242,390

79

414,130

300

NONE

345

1,204,384

320

1,120,197

107

373,399

1000

NONE

522

546,896

515

540,541

172

180,180

2200

NONE

368

175,165

367

174,709

122

58,236

200

LEADER

115

604,376

141

739,754

47

246,585

300

LEADER

186

650,280

192

670,062

64

223,354

1000

LEADER

340

356,659

328

343,808

109

114,603

2200

LEADER

310

147,846

293

139,729

98

46,576

200

ALL

74

385,594

58

304,386

19

101,462

300

ALL

105

367,282

78

272,316

26

90,772

1000

ALL

203

212,400

124

130,305

41

43,435

2200

ALL

212

100,820

136

64,835

45

21,612



Some observations from the above table

·        Increasing the number of replicas when request.required.acks is
none or leader only has limited impact on overall performance (additional
resources are required to replicate data but during tests this did not
impact producer throughput)

·        Compression is not shown as it was found that the data generated
for the test is not realistic to a production workload.  (GZIP compressed
data 300:1 which is unrealistic )

·        For some reason a message size of 1000 bytes performed the best.
Need to look into this more.


Thanks

Bert

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message