kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: Kafka 0.7 performance compared to bare metal
Date Fri, 30 Aug 2013 17:16:21 GMT
It is hard to say where the bottleneck is just from your description. Would
it be possible for you to rerun the consumer test using hprof on the
consumer so we can understand whether the fetcher is waiting on the fetches
(i.e. the broker is the bottleneck) or on the enque (i.e. the consumer is
the bottleneck). Likewise for the producer test it would be good to do the
same for the broker and producer process to understand what is happening
there.

-Jay


On Fri, Aug 30, 2013 at 1:34 AM, Rafael Bagmanov <bugzmanov@gmail.com>wrote:

> Hi,
>
> I am trying to understand how fast is kafka 0.7 compared to what I can get
> from hard drive. In essence I have 3 questions.
>
> In all tests below, I'm using single broker with single one-partitioned
> topic. Kafka perf tests have been run in 2 deployment configs:
> - broker, perf-test on same host
> - broker, perf-test on different hosts (the results are practically the
> same, so wont post them here)
>
>
> I'm using FIO(http://freecode.com/projects/fio) to benchmark speed of hard
> drives.
>
> Hardware I'm using:
> 1) m1.xlarge with ephemeral storage, 4 core cpu, 16 GB ram
> 2) hi1.4xlarge  with SSD, 16 core cpu, 64 GB ram
> 3) desktop machine with 7200 rpm sata, 4 core cpu, 8 GB ram
>
> Kafka broker config:
> Oracle jdk 1.6.0_38,  -Xmx2048
>
> socket.send.buffer=16777216
> socket.receive.buffer=16777216
> max.socket.request.bytes=104857600
> log.flush.interval=10000
> log.default.flush.interval.ms=1000
> log.default.flush.scheduler.interval.ms=1000
> num.threads=[num of cores]
>
>
> For kafka-producer-perf-test I'm assuming that IO access pattern is
> sequential write.
>
> Here is the test I ran with FIO:
>
> [sequential-write]
> rw=write
> size=50G
> ioengine=sync
> numjobs=1
> directory=/tmp/fio
> filename=redo01.log
>
>
> Here is kafka performance test:
>
> ./bin/kafka-producer-perf-test.sh -topic "perf" --batch-size 3000
> --messages 50000000 --message-size 1300 --brokerinfo
> broker.list=0:host:9092 --threads [number-of-cores]
>
>
> ----------------------------------------------------------------------------------------
> |           |   m1.xlarge            |    hi1.4xlarge       |  desktop
>  |
>
>  ----------------------------------------------------------------------------------------
> |  kafka  |     41 MB/s           |      217 MB/s       |     42 MB/s   |
>
>  -----------------------------------------------------------------------------------------
> |  fio      |     106 MB/s          |      377 MB/s       |    74 MB/s   |
>
> ----------------------------------------------------------------------------------------
>
>
> Question 1: The proportion (~1/2) is pretty stable against different kind
> of hardware I've tried. Is it as expected? Can something be done to improve
> this?
>
> I've tried to play with:
> log.flush.interval=10000
> log.default.flush.interval.ms=1000
> log.default.flush.scheduler.interval.ms=1000
>
> Like increasing 10 times, or decreasing 10 times, but haven't seen much of
> a difference in IO  throughput
>
> The other thing that bugs me much more is that kafka consumer speed on cold
> IO cache is like 5-50 times slower from what I can get with "sequential
> read" fio test.
>
> For kafka-consumer-perf-test I'm assuming that IO access pattern is
> sequential read.
>
> Here is FIO test:
>
> [sequential-read]
> rw=read
> size=50G
> ioengine=sync   # I know that kafka use sendfile, but sync should be
> slower, right?
> numjobs=1
>  directory=/tmp/fio
> filename=redo01.log
>
> Here what I'm doing with kafka-consumer-perf-test:
>
> kafka-consumer-perf-test.sh -topic "perf" --messages 50000000 --zookeeper
> host:2181 --threads 1 --socket-buffer-size 16777216 --fetch-size 16777216
>
> The broker config is  the same.
>
> I'm dropping IO cache before running tests: echo 3 >
> /proc/sys/vm/drop_caches
>
>
> -----------------------------------------------------------------------------------------------
> |           |   m1.xlarge            |    hi1.4xlarge              |
>  desktop    |
>
>  ---------------------------------------------------------------------------------------------
> |  kafka  |    25   MB/s           |     10  MB/s   (???)    |   20  MB/s
>  |
>
>  ---------------------------------------------------------------------------------------------
> |  fio      |   130   MB/s          |     450  MB/s             |    67
> MB/s  |
>
> ----------------------------------------------------------------------------------------------
>
> Question 2: Can something be done to improve consumer performance?
>
> Question 3 (most improtant for me): What might be the reasons for consumer
> to behave so badly on fastest hardware available? I see in iostat, that
> consumer really does very little read requests to hard drive
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> xvdb              0.00     0.00  144.00    0.00  6144.00     0.00    85.33
>     0.06    0.42    0.42    0.00   0.08   1.20
>
> And cpus are idling
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>            2.16    0.00    0.09    0.06    0.03   97.66
>
>
> Besides that, even if the whole topic is in IO cache, the consumer speed is
> about 45 MB/s which is still quite below my expectations.
>
> And the picture doesn't change in different deployment configs (broker and
> test on same node or 2 different nodes)
>
> Any ideas why this might happen?
>
> Rafael Bagmanov,
> Grid Dynamics.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message