spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ari Gesher (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-19764) Executors hang with supposedly running task that are really finished.
Date Wed, 01 Mar 2017 06:04:45 GMT

    [ https://issues.apache.org/jira/browse/SPARK-19764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889564#comment-15889564
] 

Ari Gesher commented on SPARK-19764:
------------------------------------

Nothing like that.  Full logs in the attached tarball.  Here's the stack trace from a stuck
executor:

{noformat}
Full thread dump OpenJDK 64-Bit Server VM (25.121-b13 mixed mode):

"shuffle-server-7" #36 daemon prio=5 os_prio=0 tid=0x00007f0764019800 nid=0xcd7 runnable [0x00007f0720684000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000c0014950> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000c00169d0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000c00148a8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	at java.lang.Thread.run(Thread.java:745)

"shuffle-server-6" #35 daemon prio=5 os_prio=0 tid=0x00007f0764017800 nid=0xb6a runnable [0x00007f07218f7000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000c0017198> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000c0019218> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000c0017100> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	at java.lang.Thread.run(Thread.java:745)

"shuffle-server-5" #34 daemon prio=5 os_prio=0 tid=0x00007f0764015800 nid=0xb69 runnable [0x00007f07219f8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000c00199e0> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000c01eee80> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000c0019948> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	at java.lang.Thread.run(Thread.java:745)

"shuffle-server-4" #33 daemon prio=5 os_prio=0 tid=0x00007f0764014000 nid=0xb1f runnable [0x00007f0721af9000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000c01ef648> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000c01f16c8> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000c01ef5b0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	at java.lang.Thread.run(Thread.java:745)

"shuffle-server-3" #32 daemon prio=5 os_prio=0 tid=0x00007f0764012000 nid=0xb0d runnable [0x00007f0721786000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000c01f1e90> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000c01f3f10> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000c01f1df8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	at java.lang.Thread.run(Thread.java:745)

"shuffle-server-2" #31 daemon prio=5 os_prio=0 tid=0x00007f0764010800 nid=0xb06 runnable [0x00007f0722bfb000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000c01f46d8> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000c022e768> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000c01f4640> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	at java.lang.Thread.run(Thread.java:745)

"threadDeathWatcher-2-1" #58 daemon prio=1 os_prio=0 tid=0x00007f0718010000 nid=0xab4 waiting
on condition [0x00007f0723efd000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:745)

"shuffle-server-1" #30 daemon prio=5 os_prio=0 tid=0x00007f076400f000 nid=0xab2 runnable [0x00007f0723ffe000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000c022ef30> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000c0230fb0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000c022ee98> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	at java.lang.Thread.run(Thread.java:745)

"qtp1090158868-57" #57 daemon prio=5 os_prio=0 tid=0x00007f0788122000 nid=0xaae waiting on
condition [0x00007f07b83d5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c0013b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
	at java.lang.Thread.run(Thread.java:745)

"qtp1090158868-56" #56 daemon prio=5 os_prio=0 tid=0x00007f0788120000 nid=0xaad waiting on
condition [0x00007f07b84d6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c0013b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
	at java.lang.Thread.run(Thread.java:745)

"qtp1090158868-55" #55 daemon prio=5 os_prio=0 tid=0x00007f078811e800 nid=0xaac waiting on
condition [0x00007f07b85d7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c0013b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
	at java.lang.Thread.run(Thread.java:745)

"qtp1090158868-54-acceptor-0@1b0de413-ServerConnector@15ca0902{HTTP/1.1}{172.31.20.156:6066}"
#54 daemon prio=5 os_prio=0 tid=0x00007f078811c800 nid=0xaab runnable [0x00007f07b86d8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
	- locked <0x00000000c0023068> (a java.lang.Object)
	at org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:377)
	at org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:745)

"qtp1090158868-53-selector-ServerConnectorManager@55b255e7/3" #53 daemon prio=5 os_prio=0
tid=0x00007f078811b000 nid=0xaaa runnable [0x00007f07b87d9000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000c002b020> (a sun.nio.ch.Util$3)
	- locked <0x00000000c002b030> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000c002afd8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601)
	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550)
	at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:745)

"qtp1090158868-52-selector-ServerConnectorManager@55b255e7/2" #52 daemon prio=5 os_prio=0
tid=0x00007f0788119000 nid=0xaa9 runnable [0x00007f07b88da000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000c0033010> (a sun.nio.ch.Util$3)
	- locked <0x00000000c0033020> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000c0032fc8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601)
	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550)
	at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:745)

"qtp1090158868-51-selector-ServerConnectorManager@55b255e7/1" #51 daemon prio=5 os_prio=0
tid=0x00007f0788117000 nid=0xaa8 runnable [0x00007f07b8adc000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000c001b1e8> (a sun.nio.ch.Util$3)
	- locked <0x00000000c001b1f8> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000c001b1a0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601)
	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550)
	at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:745)

"qtp1090158868-50-selector-ServerConnectorManager@55b255e7/0" #50 daemon prio=5 os_prio=0
tid=0x00007f0788117800 nid=0xaa7 runnable [0x00007f07b89db000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000c001adb0> (a sun.nio.ch.Util$3)
	- locked <0x00000000c001adc0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000c001ad68> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601)
	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550)
	at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:745)

"master-forward-message-thread" #48 daemon prio=5 os_prio=0 tid=0x00007f07880a6000 nid=0xaa3
waiting on condition [0x00007f07b8bdd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c0033220> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"MasterUI-47" #47 daemon prio=5 os_prio=0 tid=0x00007f078809c000 nid=0xaa2 waiting on condition
[0x00007f07b8cde000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c0033480> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
	at java.lang.Thread.run(Thread.java:745)

"MasterUI-46" #46 daemon prio=5 os_prio=0 tid=0x00007f078809a800 nid=0xaa1 waiting on condition
[0x00007f07b8ddf000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c0033480> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
	at java.lang.Thread.run(Thread.java:745)

"MasterUI-45" #45 daemon prio=5 os_prio=0 tid=0x00007f0788099000 nid=0xaa0 waiting on condition
[0x00007f07b8ee0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c0033480> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
	at java.lang.Thread.run(Thread.java:745)

"MasterUI-44-acceptor-0@21c02ae7-ServerConnector@2f07d0af{HTTP/1.1}{172.31.20.156:8080}" #44
daemon prio=5 os_prio=0 tid=0x00007f0788097000 nid=0xa9f runnable [0x00007f07b8fe1000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
	- locked <0x00000000c0033ac0> (a java.lang.Object)
	at org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:377)
	at org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:745)

"MasterUI-43-selector-ServerConnectorManager@2bbaf8f7/3" #43 daemon prio=5 os_prio=0 tid=0x00007f0788095800
nid=0xa9e runnable [0x00007f07b90e2000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000c0033da8> (a sun.nio.ch.Util$3)
	- locked <0x00000000c0033db8> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000c0033d60> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601)
	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550)
	at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:745)

"MasterUI-42-selector-ServerConnectorManager@2bbaf8f7/2" #42 daemon prio=5 os_prio=0 tid=0x00007f078808f800
nid=0xa9d runnable [0x00007f07b91e3000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000c002b2c0> (a sun.nio.ch.Util$3)
	- locked <0x00000000c002b2d0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000c002b278> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601)
	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550)
	at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:745)

"MasterUI-41-selector-ServerConnectorManager@2bbaf8f7/1" #41 daemon prio=5 os_prio=0 tid=0x00007f078808a800
nid=0xa9c runnable [0x00007f07b92e4000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000c002b560> (a sun.nio.ch.Util$3)
	- locked <0x00000000c002b570> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000c002b518> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601)
	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550)
	at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:745)

"MasterUI-40-selector-ServerConnectorManager@2bbaf8f7/0" #40 daemon prio=5 os_prio=0 tid=0x00007f078808a000
nid=0xa9b runnable [0x00007f07b93e5000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000c001b488> (a sun.nio.ch.Util$3)
	- locked <0x00000000c001b498> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000c001b440> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601)
	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550)
	at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:745)

"netty-rpc-env-timeout" #38 daemon prio=5 os_prio=0 tid=0x00007f07f123b800 nid=0xa90 waiting
on condition [0x00007f07b94e6000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c001b698> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"shuffle-server-0" #29 daemon prio=5 os_prio=0 tid=0x00007f07f11dc800 nid=0xa8e runnable [0x00007f07b97e7000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000c00141e8> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000c0014208> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000c00141a0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-7" #20 daemon prio=5 os_prio=0 tid=0x00007f07f0ffc800 nid=0xa89 waiting
on condition [0x00007f07b9cfb000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-6" #19 daemon prio=5 os_prio=0 tid=0x00007f07f0ffb000 nid=0xa88 waiting
on condition [0x00007f07b9dfc000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-5" #18 daemon prio=5 os_prio=0 tid=0x00007f07f0ff9000 nid=0xa85 waiting
on condition [0x00007f07b9efd000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-4" #17 daemon prio=5 os_prio=0 tid=0x00007f07f0ff7800 nid=0xa83 waiting
on condition [0x00007f07b9ffe000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-3" #16 daemon prio=5 os_prio=0 tid=0x00007f07f0ff6000 nid=0xa82 waiting
on condition [0x00007f07c01f6000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-2" #15 daemon prio=5 os_prio=0 tid=0x00007f07f0fcd800 nid=0xa80 waiting
on condition [0x00007f07c02f7000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-1" #14 daemon prio=5 os_prio=0 tid=0x00007f07f0fcc000 nid=0xa7e waiting
on condition [0x00007f07c03f8000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-0" #13 daemon prio=5 os_prio=0 tid=0x00007f07f0fcb800 nid=0xa7c waiting
on condition [0x00007f07c08fb000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f07f00d2000 nid=0xa6f runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f07f00c5800 nid=0xa6c waiting on
condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f07f00c1000 nid=0xa6b waiting on
condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f07f00bf000 nid=0xa6a waiting on
condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f07f00bc000 nid=0xa69 waiting on
condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f07f00ba000 nid=0xa68 waiting on
condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f07f0092800 nid=0xa62 in Object.wait() [0x00007f07c37f6000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000c01e8880> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
	- locked <0x00000000c01e8880> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f07f008e000 nid=0xa60 in Object.wait()
[0x00007f07c38f7000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000c01e8ab0> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
	- locked <0x00000000c01e8ab0> (a java.lang.ref.Reference$Lock)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=0 tid=0x00007f07f0010800 nid=0xa4b waiting on condition [0x00007f07f98c2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c00e3780> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1465)
	at org.apache.spark.rpc.netty.Dispatcher.awaitTermination(Dispatcher.scala:180)
	at org.apache.spark.rpc.netty.NettyRpcEnv.awaitTermination(NettyRpcEnv.scala:273)
	at org.apache.spark.deploy.master.Master$.main(Master.scala:1012)
	at org.apache.spark.deploy.master.Master.main(Master.scala)

"VM Thread" os_prio=0 tid=0x00007f07f0086000 nid=0xa5d runnable 

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f07f0025800 nid=0xa4e runnable 

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f07f0027800 nid=0xa50 runnable 

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f07f0029000 nid=0xa52 runnable 

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f07f002b000 nid=0xa54 runnable 

"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f07f002c800 nid=0xa56 runnable 

"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f07f002e800 nid=0xa58 runnable 

"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f07f0030000 nid=0xa5a runnable 

"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f07f0032000 nid=0xa5c runnable 

"VM Periodic Task Thread" os_prio=0 tid=0x00007f07f00d4800 nid=0xa70 waiting on condition


JNI global references: 313

Heap
 PSYoungGen      total 286720K, used 150258K [0x00000000eab00000, 0x00000000feb00000, 0x0000000100000000)
  eden space 245760K, 45% used [0x00000000eab00000,0x00000000f17d2a00,0x00000000f9b00000)
  from space 40960K, 94% used [0x00000000fc300000,0x00000000fe8e9f40,0x00000000feb00000)
  to   space 40960K, 0% used [0x00000000f9b00000,0x00000000f9b00000,0x00000000fc300000)
 ParOldGen       total 264192K, used 83384K [0x00000000c0000000, 0x00000000d0200000, 0x00000000eab00000)
  object space 264192K, 31% used [0x00000000c0000000,0x00000000c516e088,0x00000000d0200000)
 Metaspace       used 25818K, capacity 26160K, committed 26368K, reserved 1073152K
  class space    used 3429K, capacity 3512K, committed 3584K, reserved 1048576K

{noformat}

> Executors hang with supposedly running task that are really finished.
> ---------------------------------------------------------------------
>
>                 Key: SPARK-19764
>                 URL: https://issues.apache.org/jira/browse/SPARK-19764
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark, Spark Core
>    Affects Versions: 2.0.2
>         Environment: Ubuntu 16.04 LTS
> OpenJDK Runtime Environment (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)
> Spark 2.0.2 - Spark Cluster Manager
>            Reporter: Ari Gesher
>         Attachments: driver-log-stderr.log, executor-2.log, netty-6153.jpg, SPARK-19764.tgz
>
>
> We've come across a job that won't finish.  Running on a six-node cluster, each of the
executors end up with 5-7 tasks that are never marked as completed.
> Here's an excerpt from the web UI:
> ||Index  ▴||ID||Attempt||Status||Locality Level||Executor ID / Host||Launch Time||Duration||Scheduler
Delay||Task Deserialization Time||GC Time||Result Serialization Time||Getting Result Time||Peak
Execution Memory||Shuffle Read Size / Records||Errors||
> |105	| 1131	| 0	| SUCCESS	|PROCESS_LOCAL	|4 / 172.31.24.171 |	2017/02/27 22:51:36 |	1.9
min |	9 ms |	4 ms |	0.7 s |	2 ms|	6 ms|	384.1 MB| 	90.3 MB / 572	| |
> |106|	1168|	0|	RUNNING	|ANY|	2 / 172.31.16.112|	2017/02/27 22:53:25|	6.5 h	|0 ms|	0 ms|
1 s	|0 ms|	0 ms|	|384.1 MB	|98.7 MB / 624 | |	
> However, the Executor reports the task as finished: 
> {noformat}
> 17/02/27 22:53:25 INFO Executor: Running task 106.0 in stage 5.0 (TID 1168)
> 17/02/27 22:55:29 INFO Executor: Finished task 106.0 in stage 5.0 (TID 1168). 2633558
bytes result sent via BlockManager)
> {noformat}
> As does the driver log:
> {noformat}
> 17/02/27 22:53:25 INFO Executor: Running task 106.0 in stage 5.0 (TID 1168)
> 17/02/27 22:55:29 INFO Executor: Finished task 106.0 in stage 5.0 (TID 1168). 2633558
bytes result sent via BlockManager)
> {noformat}
> Full log from this executor and the {{stderr}} from {{app-20170227223614-0001/2/stderr}}
attached.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message