kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 刘明敏 <diveintotomor...@gmail.com>
Subject Re: [kafka-dev] performance of async producer stops increase when broker is more than 2
Date Wed, 28 Mar 2012 02:42:11 GMT
I have 1 producer on 1 machine.

The perf of sync producer is :

1 borker:
   *20.3277(M/s)*,*213151.4441(#/s)* (sync)
2 borkers:
   *39.5995*, *415230.6606* (sync)
3 borkers:
   *58.0836*, *609050.4903*(sync)

compared with async producer :
 when 1 broker:
 20.3164 M/s,213033.3823#/s (async)

 when 2 brokers:
 37.4754 M/s,392958.1892#/s (async)

 when 3 brokers(perf stops increase)
 36.6925 M/s, 384748.5668#/s (async)

the sync producer reached 600K #/s(far beyond sync producer's 384k #/s) So
I doubt if it is the case of maxing out network or procuder.

===
and,I post the thread from google group:
https://groups.google.com/forum/?fromgroups#!forum/kafka-dev

On Wed, Mar 28, 2012 at 3:25 AM, Jay Kreps <jay.kreps@gmail.com> wrote:

> Hi,
>
> How many producers are you using and on how many machines? Also what is the
> network speed (1gbit?) Likely you are maxing out either the producer
> machine(s) or the network itself.
>
> Also, quick question, how did you come across this address? This address
> has been dead for a while now, and the new mailing list is on Apache so I
> just want to make sure we don't have a dangling pointer somewhere. Would be
> good to move the discussion to the Apache list so others can find it.
>
> -Jay
>
> On Tue, Mar 27, 2012 at 2:10 AM, mmLiu <diveintotomorrow@gmail.com> wrote:
>
> > I tested the performance of producer with
> ./bin/kafka-producer-perf-test.sh
> >
> > when 1 broker:
> > 20.3164 M/s,213033.3823#/s (async)
> >
> > when 2 brokers:
> > 37.4754 M/s,392958.1892#/s (async)
> >
> > when 3 brokers(perf stops increase)
> > 36.6925 M/s, 384748.5668#/s (async)
> >
> > when 4 brokers(even decrease):
> > 33.2244 M/s, 348383.5006 #/s (async)
> >
> >
> > Any idea why?
> >
> > FYI:the performance of sync producer increases(almost) linearly as number
> > of brokers increases.
>



-- 
Best Regards

----------------------
刘明敏 | mmLiu

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message