kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: kafka-dev - Google Groups: Message Pending [{IK_6mr_o_d-UVSoCaWcwABV4kOW3nBaU0}]
Date Wed, 28 Mar 2012 16:04:56 GMT
Ah, but what is the cpu utilization? Can you run "top" or something during
the test? My suspicion is that the producer is maybe just cpu bound.

-Jay

On Wed, Mar 28, 2012 at 3:57 AM, 刘明敏 <diveintotomorrow@gmail.com> wrote:

> 4 cpus on Procuder:
>
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 6
> model : 30
> model name : Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz
> stepping : 5
> cpu MHz : 2393.580
> cache size : 8192 KB
> physical id : 0
> siblings : 4
> core id : 0
> cpu cores : 4
> apicid : 0
> initial apicid : 0
> fpu : yes
> fpu_exception : yes
> cpuid level : 11
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36
> clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm
> constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc
> aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm
> sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
> bogomips : 4787.16
> clflush size : 64
> cache_alignment : 64
> address sizes : 36 bits physical, 48 bits virtual
> power management:
>
> processor : 1
> vendor_id : GenuineIntel
> cpu family : 6
> model : 30
> model name : Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz
> stepping : 5
> cpu MHz : 2393.580
> cache size : 8192 KB
> physical id : 0
> siblings : 4
> core id : 1
> cpu cores : 4
> apicid : 2
> initial apicid : 2
> fpu : yes
> fpu_exception : yes
> cpuid level : 11
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36
> clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm
> constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc
> aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm
> sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
> bogomips : 4787.17
> clflush size : 64
> cache_alignment : 64
> address sizes : 36 bits physical, 48 bits virtual
> power management:
>
> processor : 2
> vendor_id : GenuineIntel
> cpu family : 6
> model : 30
> model name : Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz
> stepping : 5
> cpu MHz : 2393.580
> cache size : 8192 KB
> physical id : 0
> siblings : 4
> core id : 2
> cpu cores : 4
> apicid : 4
> initial apicid : 4
> fpu : yes
> fpu_exception : yes
> cpuid level : 11
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36
> clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm
> constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc
> aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm
> sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
> bogomips : 4787.17
> clflush size : 64
> cache_alignment : 64
> address sizes : 36 bits physical, 48 bits virtual
> power management:
>
> processor : 3
> vendor_id : GenuineIntel
> cpu family : 6
> model : 30
> model name : Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz
> stepping : 5
> cpu MHz : 2393.580
> cache size : 8192 KB
> physical id : 0
> siblings : 4
> core id : 3
> cpu cores : 4
> apicid : 6
> initial apicid : 6
> fpu : yes
> fpu_exception : yes
> cpuid level : 11
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36
> clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm
> constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc
> aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm
> sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
> bogomips : 4787.17
> clflush size : 64
> cache_alignment : 64
> address sizes : 36 bits physical, 48 bits virtual
> power management:
>
> ===
> not sure if I am sending the right place this time.
>
> 2012/3/28 Jay Kreps <jay.kreps@gmail.com>
>
> > What does CPU look like on the producer?
> >
> > Also, can you please use the apache mailing list (cc'd). This google
> group
> > is  from our pre-apache days and is long dead.
> >
> > -Jay
> >
> > On Tue, Mar 27, 2012 at 7:42 PM, Kafka
> > <kafka-dev+msgappr@googlegroups.com>wrote:
> >
> > >   A message has been sent to the kafka-dev<
> > http://groups.google.com/group/kafka-dev?hl=en_US>group and is awaiting
> > approval. We've included this message for your review.
> > >
> > > The message requires moderation because the user does not have
> permission
> > > to post.
> > >
> > > You can approve or reject<
> > http://groups.google.com/group/kafka-dev/pendmsg?hl=en_US>this message
> or
> > you can approve this message by replying to this email.
> > >  [image: Visit Google Groups]
> > >
> > > Start <https://groups.google.com/groups/create?hl=en_US&src=email>
> your
> > > own group, visit <http://groups.google.com/support/?hl=en_US&src=email
> >the
> > help center, or
> > > report <https://groups.google.com/groups/abuse?hl=en_US&src=email>
> > abuse.
> > >
> > >
> > > ---------- Forwarded message ----------
> > > From: 刘明敏 <diveintotomorrow@gmail.com>
> > > To: kafka-users@incubator.apache.org
> > > Cc: kafka-dev@googlegroups.com
> > > Date: Wed, 28 Mar 2012 10:42:11 +0800
> > > Subject: Re: [kafka-dev] performance of async producer stops increase
> > when
> > > broker is more than 2
> > > I have 1 producer on 1 machine.
> > >
> > > The perf of sync producer is :
> > >
> > > 1 borker:
> > >    *20.3277(M/s)*,*213151.4441(#/s)* (sync)
> > > 2 borkers:
> > >    *39.5995*, *415230.6606* (sync)
> > > 3 borkers:
> > >    *58.0836*, *609050.4903*(sync)
> > >
> > > compared with async producer :
> > >  when 1 broker:
> > >  20.3164 M/s,213033.3823#/s (async)
> > >
> > >  when 2 brokers:
> > >  37.4754 M/s,392958.1892#/s (async)
> > >
> > >  when 3 brokers(perf stops increase)
> > >  36.6925 M/s, 384748.5668#/s (async)
> > >
> > > the sync producer reached 600K #/s(far beyond sync producer's 384k #/s)
> > So
> > > I doubt if it is the case of maxing out network or procuder.
> > >
> > > ===
> > > and,I post the thread from google group:
> > > https://groups.google.com/forum/?fromgroups#!forum/kafka-dev
> > >
> > > On Wed, Mar 28, 2012 at 3:25 AM, Jay Kreps <jay.kreps@gmail.com>
> wrote:
> > >
> > >> Hi,
> > >>
> > >> How many producers are you using and on how many machines? Also what
> is
> > >> the
> > >> network speed (1gbit?) Likely you are maxing out either the producer
> > >> machine(s) or the network itself.
> > >>
> > >> Also, quick question, how did you come across this address? This
> address
> > >> has been dead for a while now, and the new mailing list is on Apache
> so
> > I
> > >> just want to make sure we don't have a dangling pointer somewhere.
> Would
> > >> be
> > >> good to move the discussion to the Apache list so others can find it.
> > >>
> > >> -Jay
> > >>
> > >> On Tue, Mar 27, 2012 at 2:10 AM, mmLiu <diveintotomorrow@gmail.com>
> > >> wrote:
> > >>
> > >> > I tested the performance of producer with
> > >> ./bin/kafka-producer-perf-test.sh
> > >> >
> > >> > when 1 broker:
> > >> > 20.3164 M/s,213033.3823#/s (async)
> > >> >
> > >> > when 2 brokers:
> > >> > 37.4754 M/s,392958.1892#/s (async)
> > >> >
> > >> > when 3 brokers(perf stops increase)
> > >> > 36.6925 M/s, 384748.5668#/s (async)
> > >> >
> > >> > when 4 brokers(even decrease):
> > >> > 33.2244 M/s, 348383.5006 #/s (async)
> > >> >
> > >> >
> > >> > Any idea why?
> > >> >
> > >> > FYI:the performance of sync producer increases(almost) linearly as
> > >> number
> > >> > of brokers increases.
> > >>
> > >
> > >
> > >
> > > --
> > > Best Regards
> > >
> > > ----------------------
> > > 刘明敏 | mmLiu
> > >
> > >
> > >
> >
>
>
>
> --
> Best Regards
>
> ----------------------
> 刘明敏 | mmLiu
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message