drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avi Haleva <avi.hal...@aternity.com>
Subject RE: Drill CPU Usage
Date Mon, 15 Aug 2016 13:03:03 GMT
I'm using version 1.7.0 , I guess I need to apply to patch to test this fix.
I haven't seen any reference to CPU usage in the bug report, am I missing something ?

-----Original Message-----
From: Oscar Morante [mailto:spacepluk@gmail.com] 
Sent: Monday, August 15, 2016 13:54
To: user@drill.apache.org
Subject: Re: Drill CPU Usage

Have you tried this?

https://github.com/apache/drill/commit/48b3fd2ee040f30be3ff3172bf03111ca58d4b48


On Thu, Aug 11, 2016 at 11:13:10AM +0000, Avi Haleva wrote:
>Embedding the thread dump content in the message :
>
>Thread dump 1:
>
>2016-08-11 11:02:19
>Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode):
>
>"Attach Listener" #1308 daemon prio=9 os_prio=0 tid=0x00007f4bfc156800 nid=0x1cce6 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"process reaper" #1307 daemon prio=10 os_prio=0 tid=0x00007f4c4d5cd800 nid=0x2490 runnable
[0x00007f4c000b6000]
>   java.lang.Thread.State: RUNNABLE
>	at java.lang.UNIXProcess.waitForProcessExit(Native Method)
>	at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:290)
>	at java.lang.UNIXProcess$$Lambda$7/2030562336.run(Unknown Source)
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- <0x00000006cdb44650> (a 
>java.util.concurrent.ThreadPoolExecutor$Worker)
>
>"threadDeathWatcher-2-1" #29 daemon prio=1 os_prio=0 tid=0x00007f4c4d52f800 nid=0x1dc04
waiting on condition [0x00007f4bed9d6000]
>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>	at java.lang.Thread.sleep(Native Method)
>	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- None
>
>"Client-1" #27 daemon prio=10 os_prio=0 tid=0x00007f4c4d493000 nid=0x1dc03 runnable [0x00007f4bedad7000]
>   java.lang.Thread.State: RUNNABLE
>	at io.netty.channel.epoll.Native.epollWait0(Native Method)
>	at io.netty.channel.epoll.Native.epollWait(Native.java:148)
>	at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180)
>	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205)
>	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- None
>
>"Curator-ServiceCache-0" #26 daemon prio=5 os_prio=0 tid=0x00007f4c4d552800 nid=0x1dc01
waiting on condition [0x00007f4c005c1000]
>   java.lang.Thread.State: WAITING (parking)
>	at sun.misc.Unsafe.park(Native Method)
>	- parking to wait for  <0x00000006c0013c68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- None
>
>"Curator-Framework-0" #25 daemon prio=5 os_prio=0 tid=0x00007f4c4d381000 nid=0x1dc00 waiting
on condition [0x00007f4c006c2000]
>   java.lang.Thread.State: WAITING (parking)
>	at sun.misc.Unsafe.park(Native Method)
>	- parking to wait for  <0x00000006c0023678> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>	at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
>	at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
>	at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780)
>	at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)
>	at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)
>	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- <0x00000006c00237b8> (a 
>java.util.concurrent.ThreadPoolExecutor$Worker)
>
>"main-EventThread" #24 daemon prio=5 os_prio=0 tid=0x00007f4c4d35e800 nid=0x1dbff waiting
on condition [0x00007f4c007c3000]
>   java.lang.Thread.State: WAITING (parking)
>	at sun.misc.Unsafe.park(Native Method)
>	- parking to wait for  <0x00000006c001bff8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>	at 
>org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>
>   Locked ownable synchronizers:
>	- None
>
>"main-SendThread(10.35.80.60:2181)" #23 daemon prio=5 os_prio=0 tid=0x00007f4c4d420000
nid=0x1dbfe runnable [0x00007f4c008c4000]
>   java.lang.Thread.State: RUNNABLE
>	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>	- locked <0x00000006c002b708> (a sun.nio.ch.Util$2)
>	- locked <0x00000006c002b718> (a java.util.Collections$UnmodifiableSet)
>	- locked <0x00000006c002b6c0> (a sun.nio.ch.EPollSelectorImpl)
>	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
>	at 
>org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
>
>   Locked ownable synchronizers:
>	- None
>
>"Curator-ConnectionStateManager-0" #22 daemon prio=5 os_prio=0 tid=0x00007f4c4d32c800
nid=0x1dbfd waiting on condition [0x00007f4c009c5000]
>   java.lang.Thread.State: WAITING (parking)
>	at sun.misc.Unsafe.park(Native Method)
>	- parking to wait for  <0x00000006c0033630> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245)
>	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
>	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)
>	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- <0x00000006c0033750> (a 
>java.util.concurrent.ThreadPoolExecutor$Worker)
>
>"NonBlockingInputStreamThread" #20 daemon prio=5 os_prio=0 tid=0x00007f4c4cdab800 nid=0x1dbfc
in Object.wait() [0x00007f4c01b52000]
>   java.lang.Thread.State: WAITING (on object monitor)
>	at java.lang.Object.wait(Native Method)
>	- waiting on <0x00000006c003b5f0> (a jline.internal.NonBlockingInputStream)
>	at jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278)
>	- locked <0x00000006c003b5f0> (a jline.internal.NonBlockingInputStream)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- None
>
>"Service Thread" #17 daemon prio=9 os_prio=0 tid=0x00007f4c4c10d000 nid=0x1dbe9 runnable
[0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f4c4c10a000 nid=0x1dbe8 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f4c4c108000 nid=0x1dbe7 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C1 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f4c4c105800 nid=0x1dbe6 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C1 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f4c4c103800 nid=0x1dbe5 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f4c4c101000 nid=0x1dbe4 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ff000 nid=0x1dbe3 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fd000 nid=0x1dbe2 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fb000 nid=0x1dbe1 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f8800 nid=0x1dbe0 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f6800 nid=0x1dbdf waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ec800 nid=0x1dbde waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f4c4c0df000 nid=0x1dbdd waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f4c4c0dd000 nid=0x1dbdc runnable
[0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f4c4c0b3000 nid=0x1dbdb in Object.wait()
[0x00007f4c036f5000]
>   java.lang.Thread.State: WAITING (on object monitor)
>	at java.lang.Object.wait(Native Method)
>	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>	- locked <0x00000006c0053668> (a java.lang.ref.ReferenceQueue$Lock)
>	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
>   Locked ownable synchronizers:
>	- None
>
>"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f4c4c0a9000 nid=0x1dbda in
Object.wait() [0x00007f4c037f6000]
>   java.lang.Thread.State: WAITING (on object monitor)
>	at java.lang.Object.wait(Native Method)
>	at java.lang.Object.wait(Object.java:502)
>	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
>	- locked <0x00000006c004c0d0> (a java.lang.ref.Reference$Lock)
>	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
>
>   Locked ownable synchronizers:
>	- None
>
>"main" #1 prio=5 os_prio=0 tid=0x00007f4c4c010800 nid=0x1dbcb runnable [0x00007f4c551a8000]
>   java.lang.Thread.State: RUNNABLE
>	at java.io.FileInputStream.readBytes(Native Method)
>	at java.io.FileInputStream.read(FileInputStream.java:255)
>	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>	at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
>	- locked <0x000000078ed7ef50> (a java.lang.UNIXProcess$ProcessPipeInputStream)
>	at jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:191)
>	at jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:173)
>	at jline.internal.TerminalLineSettings.stty(TerminalLineSettings.java:168)
>	at jline.internal.TerminalLineSettings.set(TerminalLineSettings.java:76)
>	at jline.UnixTerminal.disableInterruptCharacter(UnixTerminal.java:109)
>	at jline.console.ConsoleReader.readLine(ConsoleReader.java:2194)
>	at jline.console.ConsoleReader.readLine(ConsoleReader.java:2126)
>	at sqlline.SqlLine.begin(SqlLine.java:621)
>	at sqlline.SqlLine.start(SqlLine.java:375)
>	at sqlline.SqlLine.main(SqlLine.java:268)
>
>   Locked ownable synchronizers:
>	- None
>
>"VM Thread" os_prio=0 tid=0x00007f4c4c09f000 nid=0x1dbd9 runnable
>
>"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f4c4c025800 
>nid=0x1dbcc runnable
>
>"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f4c4c027800 
>nid=0x1dbcd runnable
>
>"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f4c4c029000 
>nid=0x1dbce runnable
>
>"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02b000 
>nid=0x1dbcf runnable
>
>"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02c800 
>nid=0x1dbd0 runnable
>
>"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02e800 
>nid=0x1dbd1 runnable
>
>"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f4c4c030800 
>nid=0x1dbd2 runnable
>
>"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f4c4c032000 
>nid=0x1dbd3 runnable
>
>"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f4c4c034000 
>nid=0x1dbd4 runnable
>
>"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f4c4c036000 
>nid=0x1dbd5 runnable
>
>"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f4c4c037800 
>nid=0x1dbd6 runnable
>
>"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f4c4c039800 
>nid=0x1dbd7 runnable
>
>"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f4c4c03b000 
>nid=0x1dbd8 runnable
>
>"VM Periodic Task Thread" os_prio=0 tid=0x00007f4c4c10f800 nid=0x1dbea 
>waiting on condition
>
>JNI global references: 314
>
>
>Thread dump 2:
>2016-08-11 11:02:22
>Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode):
>
>"Attach Listener" #1308 daemon prio=9 os_prio=0 tid=0x00007f4bfc156800 nid=0x1cce6 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"process reaper" #1307 daemon prio=10 os_prio=0 tid=0x00007f4c4d5cd800 nid=0x2490 runnable
[0x00007f4c000b6000]
>   java.lang.Thread.State: RUNNABLE
>	at java.io.FileInputStream.available(Native Method)
>	at java.lang.UNIXProcess$ProcessPipeInputStream.drainInputStream(UNIXProcess.java:519)
>	at java.lang.UNIXProcess$ProcessPipeInputStream.processExited(UNIXProcess.java:533)
>	- locked <0x0000000792eb6e40> (a java.lang.Object)
>	- locked <0x0000000792eb4d90> (a java.lang.UNIXProcess$ProcessPipeInputStream)
>	at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:299)
>	at java.lang.UNIXProcess$$Lambda$7/2030562336.run(Unknown Source)
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- <0x00000006cdb44650> (a 
>java.util.concurrent.ThreadPoolExecutor$Worker)
>
>"threadDeathWatcher-2-1" #29 daemon prio=1 os_prio=0 tid=0x00007f4c4d52f800 nid=0x1dc04
waiting on condition [0x00007f4bed9d6000]
>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>	at java.lang.Thread.sleep(Native Method)
>	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- None
>
>"Client-1" #27 daemon prio=10 os_prio=0 tid=0x00007f4c4d493000 nid=0x1dc03 runnable [0x00007f4bedad7000]
>   java.lang.Thread.State: RUNNABLE
>	at io.netty.channel.epoll.Native.epollWait0(Native Method)
>	at io.netty.channel.epoll.Native.epollWait(Native.java:148)
>	at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180)
>	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205)
>	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- None
>
>"Curator-ServiceCache-0" #26 daemon prio=5 os_prio=0 tid=0x00007f4c4d552800 nid=0x1dc01
waiting on condition [0x00007f4c005c1000]
>   java.lang.Thread.State: WAITING (parking)
>	at sun.misc.Unsafe.park(Native Method)
>	- parking to wait for  <0x00000006c0013c68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- None
>
>"Curator-Framework-0" #25 daemon prio=5 os_prio=0 tid=0x00007f4c4d381000 nid=0x1dc00 waiting
on condition [0x00007f4c006c2000]
>   java.lang.Thread.State: WAITING (parking)
>	at sun.misc.Unsafe.park(Native Method)
>	- parking to wait for  <0x00000006c0023678> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>	at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
>	at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
>	at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780)
>	at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)
>	at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)
>	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- <0x00000006c00237b8> (a 
>java.util.concurrent.ThreadPoolExecutor$Worker)
>
>"main-EventThread" #24 daemon prio=5 os_prio=0 tid=0x00007f4c4d35e800 nid=0x1dbff waiting
on condition [0x00007f4c007c3000]
>   java.lang.Thread.State: WAITING (parking)
>	at sun.misc.Unsafe.park(Native Method)
>	- parking to wait for  <0x00000006c001bff8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>	at 
>org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>
>   Locked ownable synchronizers:
>	- None
>
>"main-SendThread(10.35.80.60:2181)" #23 daemon prio=5 os_prio=0 tid=0x00007f4c4d420000
nid=0x1dbfe runnable [0x00007f4c008c4000]
>   java.lang.Thread.State: RUNNABLE
>	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>	- locked <0x00000006c002b708> (a sun.nio.ch.Util$2)
>	- locked <0x00000006c002b718> (a java.util.Collections$UnmodifiableSet)
>	- locked <0x00000006c002b6c0> (a sun.nio.ch.EPollSelectorImpl)
>	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
>	at 
>org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
>
>   Locked ownable synchronizers:
>	- None
>
>"Curator-ConnectionStateManager-0" #22 daemon prio=5 os_prio=0 tid=0x00007f4c4d32c800
nid=0x1dbfd waiting on condition [0x00007f4c009c5000]
>   java.lang.Thread.State: WAITING (parking)
>	at sun.misc.Unsafe.park(Native Method)
>	- parking to wait for  <0x00000006c0033630> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245)
>	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
>	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)
>	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- <0x00000006c0033750> (a 
>java.util.concurrent.ThreadPoolExecutor$Worker)
>
>"NonBlockingInputStreamThread" #20 daemon prio=5 os_prio=0 tid=0x00007f4c4cdab800 nid=0x1dbfc
in Object.wait() [0x00007f4c01b52000]
>   java.lang.Thread.State: WAITING (on object monitor)
>	at java.lang.Object.wait(Native Method)
>	- waiting on <0x00000006c003b5f0> (a jline.internal.NonBlockingInputStream)
>	at jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278)
>	- locked <0x00000006c003b5f0> (a jline.internal.NonBlockingInputStream)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- None
>
>"Service Thread" #17 daemon prio=9 os_prio=0 tid=0x00007f4c4c10d000 nid=0x1dbe9 runnable
[0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f4c4c10a000 nid=0x1dbe8 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f4c4c108000 nid=0x1dbe7 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C1 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f4c4c105800 nid=0x1dbe6 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C1 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f4c4c103800 nid=0x1dbe5 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f4c4c101000 nid=0x1dbe4 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ff000 nid=0x1dbe3 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fd000 nid=0x1dbe2 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fb000 nid=0x1dbe1 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f8800 nid=0x1dbe0 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f6800 nid=0x1dbdf waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ec800 nid=0x1dbde waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f4c4c0df000 nid=0x1dbdd waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f4c4c0dd000 nid=0x1dbdc runnable
[0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f4c4c0b3000 nid=0x1dbdb in Object.wait()
[0x00007f4c036f5000]
>   java.lang.Thread.State: WAITING (on object monitor)
>	at java.lang.Object.wait(Native Method)
>	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>	- locked <0x00000006c0053668> (a java.lang.ref.ReferenceQueue$Lock)
>	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
>   Locked ownable synchronizers:
>	- None
>
>"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f4c4c0a9000 nid=0x1dbda in
Object.wait() [0x00007f4c037f6000]
>   java.lang.Thread.State: WAITING (on object monitor)
>	at java.lang.Object.wait(Native Method)
>	at java.lang.Object.wait(Object.java:502)
>	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
>	- locked <0x00000006c004c0d0> (a java.lang.ref.Reference$Lock)
>	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
>
>   Locked ownable synchronizers:
>	- None
>
>"main" #1 prio=5 os_prio=0 tid=0x00007f4c4c010800 nid=0x1dbcb in Object.wait() [0x00007f4c551a8000]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>	at java.lang.Object.wait(Native Method)
>	at java.lang.Object.wait(Object.java:502)
>	at java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
>	- locked <0x0000000792eb2c68> (a java.lang.UNIXProcess)
>	at jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:199)
>	at jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:173)
>	at jline.internal.TerminalLineSettings.stty(TerminalLineSettings.java:168)
>	at jline.internal.TerminalLineSettings.set(TerminalLineSettings.java:76)
>	at jline.UnixTerminal.enableInterruptCharacter(UnixTerminal.java:122)
>	at jline.console.ConsoleReader.readLine(ConsoleReader.java:2859)
>	at jline.console.ConsoleReader.readLine(ConsoleReader.java:2126)
>	at sqlline.SqlLine.begin(SqlLine.java:621)
>	at sqlline.SqlLine.start(SqlLine.java:375)
>	at sqlline.SqlLine.main(SqlLine.java:268)
>
>   Locked ownable synchronizers:
>	- None
>
>"VM Thread" os_prio=0 tid=0x00007f4c4c09f000 nid=0x1dbd9 runnable
>
>"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f4c4c025800 
>nid=0x1dbcc runnable
>
>"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f4c4c027800 
>nid=0x1dbcd runnable
>
>"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f4c4c029000 
>nid=0x1dbce runnable
>
>"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02b000 
>nid=0x1dbcf runnable
>
>"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02c800 
>nid=0x1dbd0 runnable
>
>"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02e800 
>nid=0x1dbd1 runnable
>
>"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f4c4c030800 
>nid=0x1dbd2 runnable
>
>"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f4c4c032000 
>nid=0x1dbd3 runnable
>
>"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f4c4c034000 
>nid=0x1dbd4 runnable
>
>"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f4c4c036000 
>nid=0x1dbd5 runnable
>
>"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f4c4c037800 
>nid=0x1dbd6 runnable
>
>"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f4c4c039800 
>nid=0x1dbd7 runnable
>
>"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f4c4c03b000 
>nid=0x1dbd8 runnable
>
>"VM Periodic Task Thread" os_prio=0 tid=0x00007f4c4c10f800 nid=0x1dbea 
>waiting on condition
>
>JNI global references: 314
>
>Thread dump 3:
>2016-08-11 11:02:25
>Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode):
>
>"Attach Listener" #1308 daemon prio=9 os_prio=0 tid=0x00007f4bfc156800 nid=0x1cce6 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"process reaper" #1307 daemon prio=10 os_prio=0 tid=0x00007f4c4d5cd800 nid=0x2490 waiting
on condition [0x00007f4c000b6000]
>   java.lang.Thread.State: TIMED_WAITING (parking)
>	at sun.misc.Unsafe.park(Native Method)
>	- parking to wait for  <0x00000006c00435c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
>	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>	at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- None
>
>"threadDeathWatcher-2-1" #29 daemon prio=1 os_prio=0 tid=0x00007f4c4d52f800 nid=0x1dc04
waiting on condition [0x00007f4bed9d6000]
>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>	at java.lang.Thread.sleep(Native Method)
>	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- None
>
>"Client-1" #27 daemon prio=10 os_prio=0 tid=0x00007f4c4d493000 nid=0x1dc03 runnable [0x00007f4bedad7000]
>   java.lang.Thread.State: RUNNABLE
>	at io.netty.channel.epoll.Native.epollWait0(Native Method)
>	at io.netty.channel.epoll.Native.epollWait(Native.java:148)
>	at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180)
>	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205)
>	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- None
>
>"Curator-ServiceCache-0" #26 daemon prio=5 os_prio=0 tid=0x00007f4c4d552800 nid=0x1dc01
waiting on condition [0x00007f4c005c1000]
>   java.lang.Thread.State: WAITING (parking)
>	at sun.misc.Unsafe.park(Native Method)
>	- parking to wait for  <0x00000006c0013c68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- None
>
>"Curator-Framework-0" #25 daemon prio=5 os_prio=0 tid=0x00007f4c4d381000 nid=0x1dc00 waiting
on condition [0x00007f4c006c2000]
>   java.lang.Thread.State: WAITING (parking)
>	at sun.misc.Unsafe.park(Native Method)
>	- parking to wait for  <0x00000006c0023678> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>	at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
>	at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
>	at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780)
>	at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)
>	at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)
>	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- <0x00000006c00237b8> (a 
>java.util.concurrent.ThreadPoolExecutor$Worker)
>
>"main-EventThread" #24 daemon prio=5 os_prio=0 tid=0x00007f4c4d35e800 nid=0x1dbff waiting
on condition [0x00007f4c007c3000]
>   java.lang.Thread.State: WAITING (parking)
>	at sun.misc.Unsafe.park(Native Method)
>	- parking to wait for  <0x00000006c001bff8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>	at 
>org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>
>   Locked ownable synchronizers:
>	- None
>
>"main-SendThread(10.35.80.60:2181)" #23 daemon prio=5 os_prio=0 tid=0x00007f4c4d420000
nid=0x1dbfe runnable [0x00007f4c008c4000]
>   java.lang.Thread.State: RUNNABLE
>	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>	- locked <0x00000006c002b708> (a sun.nio.ch.Util$2)
>	- locked <0x00000006c002b718> (a java.util.Collections$UnmodifiableSet)
>	- locked <0x00000006c002b6c0> (a sun.nio.ch.EPollSelectorImpl)
>	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
>	at 
>org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
>
>   Locked ownable synchronizers:
>	- None
>
>"Curator-ConnectionStateManager-0" #22 daemon prio=5 os_prio=0 tid=0x00007f4c4d32c800
nid=0x1dbfd waiting on condition [0x00007f4c009c5000]
>   java.lang.Thread.State: WAITING (parking)
>	at sun.misc.Unsafe.park(Native Method)
>	- parking to wait for  <0x00000006c0033630> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245)
>	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
>	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)
>	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- <0x00000006c0033750> (a 
>java.util.concurrent.ThreadPoolExecutor$Worker)
>
>"NonBlockingInputStreamThread" #20 daemon prio=5 os_prio=0 tid=0x00007f4c4cdab800 nid=0x1dbfc
in Object.wait() [0x00007f4c01b52000]
>   java.lang.Thread.State: WAITING (on object monitor)
>	at java.lang.Object.wait(Native Method)
>	- waiting on <0x00000006c003b5f0> (a jline.internal.NonBlockingInputStream)
>	at jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278)
>	- locked <0x00000006c003b5f0> (a jline.internal.NonBlockingInputStream)
>	at java.lang.Thread.run(Thread.java:745)
>
>   Locked ownable synchronizers:
>	- None
>
>"Service Thread" #17 daemon prio=9 os_prio=0 tid=0x00007f4c4c10d000 nid=0x1dbe9 runnable
[0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f4c4c10a000 nid=0x1dbe8 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f4c4c108000 nid=0x1dbe7 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C1 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f4c4c105800 nid=0x1dbe6 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C1 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f4c4c103800 nid=0x1dbe5 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f4c4c101000 nid=0x1dbe4 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ff000 nid=0x1dbe3 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fd000 nid=0x1dbe2 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fb000 nid=0x1dbe1 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f8800 nid=0x1dbe0 waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f6800 nid=0x1dbdf waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ec800 nid=0x1dbde waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f4c4c0df000 nid=0x1dbdd waiting
on condition [0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f4c4c0dd000 nid=0x1dbdc runnable
[0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
>   Locked ownable synchronizers:
>	- None
>
>"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f4c4c0b3000 nid=0x1dbdb in Object.wait()
[0x00007f4c036f5000]
>   java.lang.Thread.State: WAITING (on object monitor)
>	at java.lang.Object.wait(Native Method)
>	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>	- locked <0x00000006c0053668> (a java.lang.ref.ReferenceQueue$Lock)
>	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
>   Locked ownable synchronizers:
>	- None
>
>"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f4c4c0a9000 nid=0x1dbda in
Object.wait() [0x00007f4c037f6000]
>   java.lang.Thread.State: WAITING (on object monitor)
>	at java.lang.Object.wait(Native Method)
>	at java.lang.Object.wait(Object.java:502)
>	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
>	- locked <0x00000006c004c0d0> (a java.lang.ref.Reference$Lock)
>	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
>
>   Locked ownable synchronizers:
>	- None
>
>"main" #1 prio=5 os_prio=0 tid=0x00007f4c4c010800 nid=0x1dbcb runnable [0x00007f4c551a8000]
>   java.lang.Thread.State: RUNNABLE
>	at java.lang.UNIXProcess.forkAndExec(Native Method)
>	at java.lang.UNIXProcess.<init>(UNIXProcess.java:248)
>	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
>	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
>	at java.lang.Runtime.exec(Runtime.java:620)
>	at java.lang.Runtime.exec(Runtime.java:485)
>	at jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:183)
>	at jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:173)
>	at jline.internal.TerminalLineSettings.stty(TerminalLineSettings.java:168)
>	at jline.internal.TerminalLineSettings.set(TerminalLineSettings.java:76)
>	at jline.UnixTerminal.enableInterruptCharacter(UnixTerminal.java:122)
>	at jline.console.ConsoleReader.readLine(ConsoleReader.java:2859)
>	at jline.console.ConsoleReader.readLine(ConsoleReader.java:2126)
>	at sqlline.SqlLine.begin(SqlLine.java:621)
>	at sqlline.SqlLine.start(SqlLine.java:375)
>	at sqlline.SqlLine.main(SqlLine.java:268)
>
>   Locked ownable synchronizers:
>	- None
>
>"VM Thread" os_prio=0 tid=0x00007f4c4c09f000 nid=0x1dbd9 runnable
>
>"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f4c4c025800 
>nid=0x1dbcc runnable
>
>"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f4c4c027800 
>nid=0x1dbcd runnable
>
>"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f4c4c029000 
>nid=0x1dbce runnable
>
>"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02b000 
>nid=0x1dbcf runnable
>
>"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02c800 
>nid=0x1dbd0 runnable
>
>"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02e800 
>nid=0x1dbd1 runnable
>
>"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f4c4c030800 
>nid=0x1dbd2 runnable
>
>"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f4c4c032000 
>nid=0x1dbd3 runnable
>
>"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f4c4c034000 
>nid=0x1dbd4 runnable
>
>"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f4c4c036000 
>nid=0x1dbd5 runnable
>
>"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f4c4c037800 
>nid=0x1dbd6 runnable
>
>"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f4c4c039800 
>nid=0x1dbd7 runnable
>
>"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f4c4c03b000 
>nid=0x1dbd8 runnable
>
>"VM Periodic Task Thread" os_prio=0 tid=0x00007f4c4c10f800 nid=0x1dbea 
>waiting on condition
>
>JNI global references: 314
>
>
>-----Original Message-----
>From: Avi Haleva [mailto:avi.haleva@aternity.com]
>Sent: Thursday, August 11, 2016 14:10
>To: user@drill.apache.org
>Subject: RE: Drill CPU Usage
>
>Both the log and out files looks clean
>
>Attached you can find the thread dump output (3 snapshots).
>I will appreciate if one can point me to the reason the drillbits 
>consume constantly CPU
>
>BTW, the CPU usage is now a constant at 40% for a single core (double 
>from a week ago)
>
>
>-----Original Message-----
>From: Dechang Gu [mailto:dgu@maprtech.com]
>Sent: Tuesday, August 2, 2016 22:21
>To: user@drill.apache.org
>Subject: Re: Drill CPU Usage
>
>On Tue, Aug 2, 2016 at 12:49 AM, Avi Haleva <avi.haleva@aternity.com> wrote:
>
>> Hi,
>> I'm in the process of evaluating Drill as a analytic repositiory.
>> I've noticed that even when idle, the drillbit process consume 20% of 
>> a single core constantly.
>>
>> Is that expected behavior or have I miss configured something.
>>
>
>I don't think that is expected. I checked my system, the drillbit CPU usage stays at 0%
almost constantly when idle.
>I did see occasionally it has a glitch at 20-30% CPU usage, due to zookeeper heartbeat
traffic, I guess.
>
>Can you collect jstack on the drillbit process to find out what is running?
>
>Also check drillbit.out and drillbit.log (typically under /var/log/drill/ ) to see if
there is anything suspicious.
>
>HTH, and Thanks,
>Dechang
>
>
>>
>> I'm running drill in single node with zookeeper in a non-embeded mode 
>> (as I want to have more than a single connection concurrently)
>>
>> Thanks,
>> Avi
>>
>>

--
Oscar Morante
"Self-education is, I firmly believe, the only kind of education there is."
                                                          -- Isaac Asimov.


Mime
View raw message