kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: slow producing on ec2
Date Tue, 10 Jan 2012 23:18:07 GMT
The majority of the problems we have seen have been due to i/o bottlenecks
on the server. iostat will uncover these, though if you aren't familiar
with it you may need to read through some docs. If iostat show high mb/sec,
high number of operations (reads or writes) per second or high service time
that is likely the issue.

-Jay

On Tue, Jan 10, 2012 at 2:55 PM, Pierre-Yves Ritschard <pyr@spootnik.org>wrote:

> will try this, thanks
>
> On Tue, Jan 10, 2012 at 11:52 PM, Jun Rao <junrao@gmail.com> wrote:
> > Pierre-Yves,
> >
> > I assume that this is a producer problem. If so, first, make sure flush
> > interval is not too small. Second, you can enable debug logging in
> > FileMessageSet to see log flush time. Then you can see if it's higher
> than
> > what you expect.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Jan 10, 2012 at 2:19 PM, Pierre-Yves Ritschard <pyr@spootnik.org
> >wrote:
> >
> >> Hi,
> >>
> >> One of my cluster runs on ec2 and I noticed that I have rather slow
> >> throughput, less than 500 per second. I isolated the trouble on the
> >> following setup
> >>
> >> 1 producer instance (m1.large)
> >> 5 consumer instances also running a broker (m2.xlarge)
> >> 1 zookeeper instance (m1.large)
> >>
> >> I am using a sync producer and have 60 partitions.
> >>
> >> On my local box, with everything running I would several thousand per
> >> second.
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message