kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Wu <stevenz...@gmail.com>
Subject Re: high cpu and network traffic when cluster has no topic
Date Wed, 04 Feb 2015 21:15:45 GMT
Bhavesh,

unfortunately, ps cmd in Mac doesn't display thread id. I tried DTrace, but
it only shows kernel thread id (not Java thread id).

anyway, I updated the jira with producer metrics. it clearly shows request
rate shoot up to 18K/sec.

Thanks,
Steven

On Wed, Feb 4, 2015 at 9:48 AM, Steven Wu <stevenz3wu@gmail.com> wrote:

> Bhavesh,
>
> this is on Mac OS. I couldn't get similar options to make ps/jstack work
> on Mac. will continue to try if I can make them work.
>
> logging output does show "kafka-producer-network-thread" sends two
> metadata requests per milli-seconds.
>
> Thanks,
> Steven
>
>
> On Wed, Feb 4, 2015 at 9:15 AM, Bhavesh Mistry <mistry.p.bhavesh@gmail.com
> > wrote:
>
>> Hi Steven,
>>
>> Can you please try to see if io thread is indeed a problem ?  The
>> following
>> on works on Linux:
>>
>> ps  -p "$java_pid" -L -o tid,pcpu
>> jstack -F "$java_pid"
>>
>> Then compare the thread # (may have to Hex # to decimal) between the
>> Jstack
>> and ps command.  This will  tell you which thread is consuming more CPU
>> for
>> that process.
>>
>> Thanks,
>>
>> Bhavesh
>>
>> On Wed, Feb 4, 2015 at 9:01 AM, Steven Wu <stevenz3wu@gmail.com> wrote:
>>
>> > I have re-run my unit test with 0.8.2.0. same tight-loop problem
>> happened
>> > after a few mins.
>> >
>> > On Tue, Feb 3, 2015 at 10:00 PM, Guozhang Wang <wangguoz@gmail.com>
>> wrote:
>> >
>> > > Steven, you may be hitting on KAFKA-1642
>> > > <https://issues.apache.org/jira/browse/KAFKA-1642>.
>> > >
>> > > As Jay said, a bunch of such issues are fixed in the new release.
>> Please
>> > > let us know if you still see the issue with it.
>> > >
>> > > Guozhang
>> > >
>> > > On Tue, Feb 3, 2015 at 8:52 PM, Steven Wu <stevenz3wu@gmail.com>
>> wrote:
>> > >
>> > > > sure. will try my unit test again with 0.8.2.0 release tomorrow and
>> > > report
>> > > > back my findings.
>> > > >
>> > > > On Tue, Feb 3, 2015 at 8:42 PM, Jay Kreps <jay.kreps@gmail.com>
>> wrote:
>> > > >
>> > > > > Hey Steven,
>> > > > >
>> > > > > That sounds like a bug. I think we fixed a few producer high
cpu
>> > issues
>> > > > > since the beta, I wonder if you could repeat the same test with
>> the
>> > > > 0.8.2.
>> > > > > final release?
>> > > > >
>> > > > > -Jay
>> > > > >
>> > > > > On Tue, Feb 3, 2015 at 8:37 PM, Steven Wu <stevenz3wu@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > actually, my local test can reproduce the issue although
not
>> > > > immediately.
>> > > > > > seems to happen after a few mins. I enabled TRACE level
logging.
>> > here
>> > > > > seems
>> > > > > > to be the tight loop. you can see that there are two metadata
>> > > requests
>> > > > in
>> > > > > > one milli-seconds.
>> > > > > >
>> > > > > > kafka-producer-network-thread | foo 20:34:32,626 TRACE
>> > > > NetworkClient:301
>> > > > > -
>> > > > > > Ignoring empty metadata response with correlation id 360185.
>> > > > > > kafka-producer-network-thread | foo 20:34:32,626 DEBUG
>> > > > NetworkClient:369
>> > > > > -
>> > > > > > Trying to send metadata request to node -2
>> > > > > > kafka-producer-network-thread | foo 20:34:32,626 DEBUG
>> > > > NetworkClient:374
>> > > > > -
>> > > > > > Sending metadata request ClientRequest(expectResponse=true,
>> > > > payload=null,
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> request=RequestSend(header={api_key=3,api_version=0,correlation_id=360186,client_id=foo},
>> > > > > > body={topics=[]})) to node -2
>> > > > > > kafka-producer-network-thread | foo 20:34:32,626 TRACE
>> > > > NetworkClient:301
>> > > > > -
>> > > > > > Ignoring empty metadata response with correlation id 360186.
>> > > > > > kafka-producer-network-thread | foo 20:34:32,626 DEBUG
>> > > > NetworkClient:369
>> > > > > -
>> > > > > > Trying to send metadata request to node -2
>> > > > > >
>> > > > > >
>> > > > > > On Tue, Feb 3, 2015 at 8:10 PM, Steven Wu <stevenz3wu@gmail.com
>> >
>> > > > wrote:
>> > > > > >
>> > > > > > > Hi,
>> > > > > > >
>> > > > > > > We have observed high cpu and high network traffic
problem
>> when
>> > > > > > > 1) cluster (0.8.1.1) has no topic
>> > > > > > > 2) KafkaProducer (0.8.2-beta) object is created without
>> sending
>> > any
>> > > > > > traffic
>> > > > > > >
>> > > > > > > We have observed such problem twice. In both cases,
problem
>> went
>> > > away
>> > > > > > > immediately after one/any topic is created.
>> > > > > > >
>> > > > > > > Is this a known issue? Just want to check with the
community
>> > first
>> > > > > before
>> > > > > > > I spend much time to reproduce it.
>> > > > > > >
>> > > > > > > I couldn't reproduce the issue with similar setup with
unit
>> test
>> > > code
>> > > > > in
>> > > > > > > IDE. start two brokers with no topic locally on my
laptop.
>> > create a
>> > > > > > > KafkaProducer object without sending any msgs. but
I only
>> tested
>> > > with
>> > > > > > > 0.8.2-beta for both broker and producer.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Steven
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > -- Guozhang
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message