kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: kafka-dev - Google Groups: Message Pending [{IK_6mr_o_d-UVSoCaWcwABV4kOW3nBaU0}]
Date Wed, 28 Mar 2012 06:34:26 GMT
What does CPU look like on the producer?

Also, can you please use the apache mailing list (cc'd). This google group
is  from our pre-apache days and is long dead.

-Jay

On Tue, Mar 27, 2012 at 7:42 PM, Kafka
<kafka-dev+msgappr@googlegroups.com>wrote:

>   A message has been sent to the kafka-dev<http://groups.google.com/group/kafka-dev?hl=en_US>group
and is awaiting approval. We've included this message for your review.
>
> The message requires moderation because the user does not have permission
> to post.
>
> You can approve or reject<http://groups.google.com/group/kafka-dev/pendmsg?hl=en_US>this
message or you can approve this message by replying to this email.
>  [image: Visit Google Groups]
>
> Start <https://groups.google.com/groups/create?hl=en_US&src=email> your
> own group, visit <http://groups.google.com/support/?hl=en_US&src=email>the
help center, or
> report <https://groups.google.com/groups/abuse?hl=en_US&src=email> abuse.
>
>
> ---------- Forwarded message ----------
> From: 刘明敏 <diveintotomorrow@gmail.com>
> To: kafka-users@incubator.apache.org
> Cc: kafka-dev@googlegroups.com
> Date: Wed, 28 Mar 2012 10:42:11 +0800
> Subject: Re: [kafka-dev] performance of async producer stops increase when
> broker is more than 2
> I have 1 producer on 1 machine.
>
> The perf of sync producer is :
>
> 1 borker:
>    *20.3277(M/s)*,*213151.4441(#/s)* (sync)
> 2 borkers:
>    *39.5995*, *415230.6606* (sync)
> 3 borkers:
>    *58.0836*, *609050.4903*(sync)
>
> compared with async producer :
>  when 1 broker:
>  20.3164 M/s,213033.3823#/s (async)
>
>  when 2 brokers:
>  37.4754 M/s,392958.1892#/s (async)
>
>  when 3 brokers(perf stops increase)
>  36.6925 M/s, 384748.5668#/s (async)
>
> the sync producer reached 600K #/s(far beyond sync producer's 384k #/s) So
> I doubt if it is the case of maxing out network or procuder.
>
> ===
> and,I post the thread from google group:
> https://groups.google.com/forum/?fromgroups#!forum/kafka-dev
>
> On Wed, Mar 28, 2012 at 3:25 AM, Jay Kreps <jay.kreps@gmail.com> wrote:
>
>> Hi,
>>
>> How many producers are you using and on how many machines? Also what is
>> the
>> network speed (1gbit?) Likely you are maxing out either the producer
>> machine(s) or the network itself.
>>
>> Also, quick question, how did you come across this address? This address
>> has been dead for a while now, and the new mailing list is on Apache so I
>> just want to make sure we don't have a dangling pointer somewhere. Would
>> be
>> good to move the discussion to the Apache list so others can find it.
>>
>> -Jay
>>
>> On Tue, Mar 27, 2012 at 2:10 AM, mmLiu <diveintotomorrow@gmail.com>
>> wrote:
>>
>> > I tested the performance of producer with
>> ./bin/kafka-producer-perf-test.sh
>> >
>> > when 1 broker:
>> > 20.3164 M/s,213033.3823#/s (async)
>> >
>> > when 2 brokers:
>> > 37.4754 M/s,392958.1892#/s (async)
>> >
>> > when 3 brokers(perf stops increase)
>> > 36.6925 M/s, 384748.5668#/s (async)
>> >
>> > when 4 brokers(even decrease):
>> > 33.2244 M/s, 348383.5006 #/s (async)
>> >
>> >
>> > Any idea why?
>> >
>> > FYI:the performance of sync producer increases(almost) linearly as
>> number
>> > of brokers increases.
>>
>
>
>
> --
> Best Regards
>
> ----------------------
> 刘明敏 | mmLiu
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message