kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Smith <esm...@stardotstar.org>
Subject Re: [kafka-dev] performance of async producer stops increase when broker is more than 2
Date Thu, 29 Mar 2012 13:14:30 GMT
I too have noticed in my initial testing that the sync producer is
faster than the async.  Still a kafka newbie, but my 2 cents.

Ed

On Tue, Mar 27, 2012 at 10:42 PM, 刘明敏 <diveintotomorrow@gmail.com> wrote:
> I have 1 producer on 1 machine.
>
> The perf of sync producer is :
>
> 1 borker:
>   *20.3277(M/s)*,*213151.4441(#/s)* (sync)
> 2 borkers:
>   *39.5995*, *415230.6606* (sync)
> 3 borkers:
>   *58.0836*, *609050.4903*(sync)
>
> compared with async producer :
>  when 1 broker:
>  20.3164 M/s,213033.3823#/s (async)
>
>  when 2 brokers:
>  37.4754 M/s,392958.1892#/s (async)
>
>  when 3 brokers(perf stops increase)
>  36.6925 M/s, 384748.5668#/s (async)
>
> the sync producer reached 600K #/s(far beyond sync producer's 384k #/s) So
> I doubt if it is the case of maxing out network or procuder.
>
> ===
> and,I post the thread from google group:
> https://groups.google.com/forum/?fromgroups#!forum/kafka-dev
>
> On Wed, Mar 28, 2012 at 3:25 AM, Jay Kreps <jay.kreps@gmail.com> wrote:
>
>> Hi,
>>
>> How many producers are you using and on how many machines? Also what is the
>> network speed (1gbit?) Likely you are maxing out either the producer
>> machine(s) or the network itself.
>>
>> Also, quick question, how did you come across this address? This address
>> has been dead for a while now, and the new mailing list is on Apache so I
>> just want to make sure we don't have a dangling pointer somewhere. Would be
>> good to move the discussion to the Apache list so others can find it.
>>
>> -Jay
>>
>> On Tue, Mar 27, 2012 at 2:10 AM, mmLiu <diveintotomorrow@gmail.com> wrote:
>>
>> > I tested the performance of producer with
>> ./bin/kafka-producer-perf-test.sh
>> >
>> > when 1 broker:
>> > 20.3164 M/s,213033.3823#/s (async)
>> >
>> > when 2 brokers:
>> > 37.4754 M/s,392958.1892#/s (async)
>> >
>> > when 3 brokers(perf stops increase)
>> > 36.6925 M/s, 384748.5668#/s (async)
>> >
>> > when 4 brokers(even decrease):
>> > 33.2244 M/s, 348383.5006 #/s (async)
>> >
>> >
>> > Any idea why?
>> >
>> > FYI:the performance of sync producer increases(almost) linearly as number
>> > of brokers increases.
>>
>
>
>
> --
> Best Regards
>
> ----------------------
> 刘明敏 | mmLiu

Mime
View raw message