kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: Is 0.7.1 slower than 0.7.0
Date Tue, 10 Jul 2012 15:16:50 GMT
There are usually three reasons things get slow:

   1. Locking--no over-utilized resource
   2. I/O--disk over-utilized (either throughput or iops)
   3. CPU

In addition since I think this is a single producer case the bottleneck may
be either the producer or the broker.

Couple of things we can do to trace this down:

   - Try adding a few more producers (or even just producer threads) to
   both tests and see if that improves throughput The perf tool should make
   this fairly easy. If so that would tend to indicate a bottleneck in the
   producer.
   - Try checking IOPS and throughput in both versions. Theoretically with
   the same config they should produce the same IOPs pattern. If not that may
   be the root cause. To get this out of your dstat add the -d and -r options.
   - Run hprof or another cpu profiler on both clients or both servers and
   compare the trace. This should show any difference in cpu usage between the
   two.

Since producer CPU is lower in your trace in 0.7.1, producer CPU can't be
the bottleneck. Disk throughput is only half in 0.7.1 but it isn't clear if
that is cause or effect. What is the flush policy (log.flush.interval and
log.default.flush.interval.ms)? What are the IOPS between the two?

I also notice that broker CPU in 0.7.1 is 2x what it is in 0.7.0. That is a
possibility. How many CPU cores does the box have? Even though it is only
8-9% if there is only one producer thread then there is only one broker
thread working to process that, so that might be the bottleneck.

-Jay

On Tue, Jul 10, 2012 at 7:45 AM, Jun Rao <junrao@gmail.com> wrote:

> Min,
>
> Interesting. Could you file a jira and attach your producer code and dstat
> output there?
>
> Thanks,
>
> Jun
>
> On Mon, Jul 9, 2012 at 11:53 PM, Min <miniway@gmail.com> wrote:
>
> > Hello,
> >
> > I tested again and I could reproduce the situation.
> >
> > I run 0.7.0 with the following shell command
> > ./kafka-0.7.0-incubating-src/bin/kafka-server-start.sh
> > config/server.properties
> > then 0.7.1
> > ./kafka-0.7.1-incubating/bin/kafka-server-start.sh
> config/server.properties
> >
> > configuration values in the server.properties are all default but the
> > ZK address.
> >
> > My producer and broker were running on different machines in same rack.
> > They are not VMs but physical machines. Also I don't feel some other
> > process interfere my test.
> >
> >
> > I'm attaching the dstat result and my producer code.
> > Producer is so simple that read files in a given directory by line and
> > send every 1000 lines to the broker.
> >
> > Thanks
> > Min
> >
> > ./kafka-0.7.0-incubating-src/bin/kafka-server-start.sh
> > config/server.properties
> > (Producer)
> > ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
> > usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
> >  23   2  72   2   0   1|  88M  896k|  98k   50M|   0     0 |2855  2627
> >  20   1  77   1   0   1|  81M 1144k|  79k   48M|   0     0 |2858  2423
> >  21   1  77   0   0   1|  85M    0 |  77k   47M|   0     0 |2655  2222
> >   6   0  93   0   0   0|  24M  240k|  38k   15M|   0     0 |1664  1232
> >  18   1  80   0   1   1|  71M    0 |  72k   42M|   0     0 |2657  2152
> >  21   1  69   7   0   1|  75M 1704k|  92k   48M|   0     0 |2959  2872
> >  20   2  52  25   0   1|  90M  512k|  82k   48M|   0     0 |2884  2940
> >  20   1  78   0   0   0|  85M    0 |  77k   47M|   0     0 |2800  2459
> >  22   2  75   0   1   1|  88M  320k|  81k   51M|   0     0 |2840  2590
> >  24   1  74   0   1   0|  90M    0 |  88k   52M|   0     0 |2819  2526
> >  23   1  71   2   0   1|  87M 1408k|  96k   51M|   0     0 |2924  2661
> >  23   2  73   1   1   1|  89M  744k|  82k   51M|   0     0 |2851  2485
> >  25   1  72   0   1   1|  94M    0 |  89k   55M|   0     0 |2941  2545
> >  22   1  76   0   0   1|  88M  240k|  81k   50M|   0     0 |2804  2581
> >  23   1  75   0   0   1|  93M    0 |  83k   53M|   0     0 |2868  2530
> >  23   1  69   5   0   1|  87M 1400k| 100k   53M|   0     0 |2959  2668
> >
> > (Broker 0.7.0)
> > usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
> >   4   3  86   4   0   2|2056k  197M|  51M   68k|   0     0 |  11k 1911
> >   5   3  86   4   0   2|2048k  209M|  53M   72k|   0     0 |  11k 2097
> >   5   3  85   4   0   2|2056k  207M|  52M   77k|   0     0 |  11k 2084
> >   4   4  86   4   0   2|2048k  208M|  53M   71k|   0     0 |  11k 2101
> >   5   3  86   4   0   2|2064k  201M|  51M   68k|   0     0 |  11k 2055
> >   4   4  86   4   0   2|2048k  207M|  53M   71k|   0     0 |  11k 2007
> >   5   4  86   4   0   2|2048k  209M|  53M   71k|   0     0 |  11k 2018
> >   5   3  85   4   0   2|2056k  207M|  52M   76k|   0     0 |  11k 1997
> >   4   4  85   5   0   2|2056k  197M|  50M   67k|   0     0 |  11k 1907
> >   5   4  86   4   0   2|2056k  205M|  52M   71k|   0     0 |  11k 2053
> >   5   3  86   4   0   2|2048k  205M|  52M   70k|   0     0 |  11k 2000
> >
> > ./kafka-0.7.1-incubating/bin/kafka-server-start.sh
> config/server.properties
> >
> > (Producer)
> > ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
> > usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
> >   9   1  89   1   0   0|  41M  416k|  51k   27M|   0     0 |2205  1618
> >   9   1  88   2   0   0|  49M    0 |  47k   27M|   0     0 |2212  1952
> >   8   1  89   1   0   0|  44M    0 |  58k   26M|   0     0 |2201  1713
> >  10   1  87   2   0   0|  47M  256k|  57k   26M|   0     0 |2199  1992
> >   9   1  84   5   0   1|  46M 2488k|  56k   27M|   0     0 |2472  1924
> >   9   1  89   1   0   0|  42M  384k|  53k   26M|   0     0 |2187  1730
> >   9   1  89   1   0   0|  48M    0 |  44k   27M|   0     0 |2189  1691
> >  10   1  88   1   0   0|  45M    0 |  58k   27M|   0     0 |2205  1884
> >   9   1  89   1   0   0|  48M  224k|  58k   27M|   0     0 |2225  1715
> >   9   1  76  12   0   1|  44M 2496k|  57k   26M|   0     0 |2425  2133
> >   9   1  88   2   0   0|  45M  584k|  51k   26M|   0     0 |2194  1685
> >   9   1  89   1   0   0|  46M    0 |  43k   25M|   0     0 |2166  1727
> >   9   1  90   1   0   0|  47M    0 |  54k   25M|   0     0 |2183  1629
> >
> > (Broker - 0.7.1)
> > ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
> > usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
> >   9   2  86   2   0   1|2048k  103M|  26M   36k|   0     0 |6376  1849
> >   8   2  86   2   0   1|2056k  104M|  26M   37k|   0     0 |6431  1945
> >   9   2  86   2   0   1|2048k  101M|  26M   42k|   0     0 |6431  1884
> >   9   2  87   2   0   1|2048k  104M|  26M   35k|   0     0 |6300  1561
> >   9   2  86   2   0   1|2048k  104M|  27M   37k|   0     0 |6542  1543
> >   9   2  86   2   0   1|2056k  104M|  26M   37k|   0     0 |6487  1638
> >   9   2  87   2   0   1|2048k  101M|  26M   36k|   0     0 |6402  1721
> >   9   2  86   2   0   1|2048k  104M|  26M   43k|   0     0 |6436  1762
> >   9   2  86   2   0   1|2048k  106M|  27M   37k|   0     0 |6527  1799
> >   9   2  86   3   0   1|2056k  101M|  26M   36k|   0     0 |6378  1521
> >   9   2  86   2   0   1|2048k  104M|  27M   37k|   0     0 |6521  1568
> >   9   2  87   2   0   1|2048k  104M|  26M   37k|   0     0 |6452  1708
> >
> >
> >
> >
> > 2012/7/9 Jay Kreps <jay.kreps@gmail.com>:
> > > Not that is known to me. Is this something you can reproduce? Are the
> > > settings the same on the two brokers? Can you check iostat between the
> > two
> > > and see if the change is due to disk activity or cpu?
> > >
> > > -Jay
> > >
> > > On Sun, Jul 8, 2012 at 12:03 AM, Min <miniway@gmail.com> wrote:
> > >
> > >> Hello,
> > >>
> > >> Recently I've upgrade to 0.7.1.  But 0.7.1 didn't show performance as
> > >> 0.7.0.
> > >>
> > >> With 0.7.0 I can push 45MBps to the broker. But I only can push 27MBps
> > >> with 0.7.1.
> > >>
> > >> Is there any a fetch to affect the performance?
> > >>
> > >> Thanks
> > >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message