kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <...@confluent.io>
Subject Re: Unreasonably high CPU from Kafka (0.8.2.1)
Date Thu, 17 Sep 2015 22:30:42 GMT
The selector is probably not the issue. If there is no incoming traffic,
selector.select(300) won't return until after 300ms.

Thanks,

Jun

On Thu, Sep 17, 2015 at 1:13 PM, Jaikiran Pai <jai.forums2013@gmail.com>
wrote:

> Sending this to the dev list since the Kafka dev team might have more
> inputs on this one. Can someone please take a look at the issue noted below
> and whether the suggested change makes sense?
>
> -Jaikiran
>
> On Tuesday 15 September 2015 12:03 AM, Jaikiran Pai wrote:
>
>> We have been using Kafka for a while now in one of dev projects.
>> Currently we have just 1 broker and 1 zookeeper instance. Almost every day,
>> Kafka "stalls" and we end up cleaning up the data/log folder of Kafka and
>> zookeeper and bring it up afresh. We haven't been able to narrow down the
>> issue yet.
>>
>> However, keeping aside that part for a while, we have been noticing that
>> even when the system/application is completely idle, the Kafka process
>> seems to take up unreasonably high CPU (10-15% constantly shown in top
>> command). We have taken multiple thread dumps and each of them have this:
>>
>> "kafka-socket-acceptor" #24 prio=5 os_prio=0 tid=0x00007f62685d9000
>> nid=0x2d47 runnable [0x00007f6231464000]
>>    java.lang.Thread.State: RUNNABLE
>>     at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>     at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>     at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>     - locked <0x00000000ca77a458> (a sun.nio.ch.Util$2)
>>     - locked <0x00000000ca77a440> (a
>> java.util.Collections$UnmodifiableSet)
>>     - locked <0x00000000ca774550> (a sun.nio.ch.EPollSelectorImpl)
>>     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>     at kafka.network.Acceptor.run(SocketServer.scala:215)
>>     at java.lang.Thread.run(Thread.java:745)
>>
>> "kafka-network-thread-9092-2" #23 prio=5 os_prio=0 tid=0x00007f62685d6800
>> nid=0x2d46 runnable [0x00007f6231565000]
>>    java.lang.Thread.State: RUNNABLE
>>     at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>     at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>     at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>     - locked <0x00000000ca77d050> (a sun.nio.ch.Util$2)
>>     - locked <0x00000000ca77d038> (a
>> java.util.Collections$UnmodifiableSet)
>>     - locked <0x00000000ca7745e0> (a sun.nio.ch.EPollSelectorImpl)
>>     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>     at kafka.network.Processor.run(SocketServer.scala:320)
>>     at java.lang.Thread.run(Thread.java:745)
>>
>> "kafka-network-thread-9092-1" #22 prio=5 os_prio=0 tid=0x00007f62685c7800
>> nid=0x2d45 runnable [0x00007f6231666000]
>>    java.lang.Thread.State: RUNNABLE
>>     at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>     at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>     at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>     - locked <0x00000000ca77e590> (a sun.nio.ch.Util$2)
>>     - locked <0x00000000ca77e578> (a
>> java.util.Collections$UnmodifiableSet)
>>     - locked <0x00000000ca7746b8> (a sun.nio.ch.EPollSelectorImpl)
>>     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>     at kafka.network.Processor.run(SocketServer.scala:320)
>>     at java.lang.Thread.run(Thread.java:745)
>>
>> "kafka-network-thread-9092-0" #21 prio=5 os_prio=0 tid=0x00007f62685b9000
>> nid=0x2d44 runnable [0x00007f6231767000]
>>    java.lang.Thread.State: RUNNABLE
>>     at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>     at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>     at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>     - locked <0x00000000ca77fbd0> (a sun.nio.ch.Util$2)
>>     - locked <0x00000000ca77fbb8> (a
>> java.util.Collections$UnmodifiableSet)
>>     - locked <0x00000000ca774790> (a sun.nio.ch.EPollSelectorImpl)
>>     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>     at kafka.network.Processor.run(SocketServer.scala:320)
>>     at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>>
>> Looking at the code of 0.8.2.1, this piece of code looks like
>> https://github.com/apache/kafka/blob/0.8.2.1/core/src/main/scala/kafka/network/SocketServer.scala#L314
>> :
>>
>> while(isRunning) {
>> ...
>>     val ready = selector.select(300)
>>     ...
>>     if(ready > 0) {
>>         ...
>>     }
>> ...
>> }
>>
>> This looks like a (always) "busy" while loop when selector.select returns
>> 0. Could a sleep for a few milli. seconds help in this case? Similar code
>> is present in the Acceptor in that same file, which does this exact thing.
>> Would adding some small sleep in there help with reducing the CPU usage
>> when things are idle?
>>
>> -Jaikiran
>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message