kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Wu <stevenz...@gmail.com>
Subject Re: high cpu and network traffic when cluster has no topic
Date Wed, 04 Feb 2015 17:48:19 GMT
Bhavesh,

this is on Mac OS. I couldn't get similar options to make ps/jstack work on
Mac. will continue to try if I can make them work.

logging output does show "kafka-producer-network-thread" sends two metadata
requests per milli-seconds.

Thanks,
Steven


On Wed, Feb 4, 2015 at 9:15 AM, Bhavesh Mistry <mistry.p.bhavesh@gmail.com>
wrote:

> Hi Steven,
>
> Can you please try to see if io thread is indeed a problem ?  The following
> on works on Linux:
>
> ps  -p "$java_pid" -L -o tid,pcpu
> jstack -F "$java_pid"
>
> Then compare the thread # (may have to Hex # to decimal) between the Jstack
> and ps command.  This will  tell you which thread is consuming more CPU for
> that process.
>
> Thanks,
>
> Bhavesh
>
> On Wed, Feb 4, 2015 at 9:01 AM, Steven Wu <stevenz3wu@gmail.com> wrote:
>
> > I have re-run my unit test with 0.8.2.0. same tight-loop problem happened
> > after a few mins.
> >
> > On Tue, Feb 3, 2015 at 10:00 PM, Guozhang Wang <wangguoz@gmail.com>
> wrote:
> >
> > > Steven, you may be hitting on KAFKA-1642
> > > <https://issues.apache.org/jira/browse/KAFKA-1642>.
> > >
> > > As Jay said, a bunch of such issues are fixed in the new release.
> Please
> > > let us know if you still see the issue with it.
> > >
> > > Guozhang
> > >
> > > On Tue, Feb 3, 2015 at 8:52 PM, Steven Wu <stevenz3wu@gmail.com>
> wrote:
> > >
> > > > sure. will try my unit test again with 0.8.2.0 release tomorrow and
> > > report
> > > > back my findings.
> > > >
> > > > On Tue, Feb 3, 2015 at 8:42 PM, Jay Kreps <jay.kreps@gmail.com>
> wrote:
> > > >
> > > > > Hey Steven,
> > > > >
> > > > > That sounds like a bug. I think we fixed a few producer high cpu
> > issues
> > > > > since the beta, I wonder if you could repeat the same test with the
> > > > 0.8.2.
> > > > > final release?
> > > > >
> > > > > -Jay
> > > > >
> > > > > On Tue, Feb 3, 2015 at 8:37 PM, Steven Wu <stevenz3wu@gmail.com>
> > > wrote:
> > > > >
> > > > > > actually, my local test can reproduce the issue although not
> > > > immediately.
> > > > > > seems to happen after a few mins. I enabled TRACE level logging.
> > here
> > > > > seems
> > > > > > to be the tight loop. you can see that there are two metadata
> > > requests
> > > > in
> > > > > > one milli-seconds.
> > > > > >
> > > > > > kafka-producer-network-thread | foo 20:34:32,626 TRACE
> > > > NetworkClient:301
> > > > > -
> > > > > > Ignoring empty metadata response with correlation id 360185.
> > > > > > kafka-producer-network-thread | foo 20:34:32,626 DEBUG
> > > > NetworkClient:369
> > > > > -
> > > > > > Trying to send metadata request to node -2
> > > > > > kafka-producer-network-thread | foo 20:34:32,626 DEBUG
> > > > NetworkClient:374
> > > > > -
> > > > > > Sending metadata request ClientRequest(expectResponse=true,
> > > > payload=null,
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> request=RequestSend(header={api_key=3,api_version=0,correlation_id=360186,client_id=foo},
> > > > > > body={topics=[]})) to node -2
> > > > > > kafka-producer-network-thread | foo 20:34:32,626 TRACE
> > > > NetworkClient:301
> > > > > -
> > > > > > Ignoring empty metadata response with correlation id 360186.
> > > > > > kafka-producer-network-thread | foo 20:34:32,626 DEBUG
> > > > NetworkClient:369
> > > > > -
> > > > > > Trying to send metadata request to node -2
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 3, 2015 at 8:10 PM, Steven Wu <stevenz3wu@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > We have observed high cpu and high network traffic problem
when
> > > > > > > 1) cluster (0.8.1.1) has no topic
> > > > > > > 2) KafkaProducer (0.8.2-beta) object is created without
sending
> > any
> > > > > > traffic
> > > > > > >
> > > > > > > We have observed such problem twice. In both cases, problem
> went
> > > away
> > > > > > > immediately after one/any topic is created.
> > > > > > >
> > > > > > > Is this a known issue? Just want to check with the community
> > first
> > > > > before
> > > > > > > I spend much time to reproduce it.
> > > > > > >
> > > > > > > I couldn't reproduce the issue with similar setup with
unit
> test
> > > code
> > > > > in
> > > > > > > IDE. start two brokers with no topic locally on my laptop.
> > create a
> > > > > > > KafkaProducer object without sending any msgs. but I only
> tested
> > > with
> > > > > > > 0.8.2-beta for both broker and producer.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Steven
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message