spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: Spark 2.0.0 job completes but hangs
Date Mon, 10 Oct 2016 15:54:49 GMT
I am pretty sure that's
https://issues.apache.org/jira/browse/SPARK-17707

On Mon, Oct 10, 2016, 16:26 jamborta <jamborta@gmail.com> wrote:

> Hi all,
>
> I have a spark job that takes about an hour to run, in the end it completes
> all the task, then the job just hangs and does nothing (it writes to s3 as
> the last step, which also gets completed, all files appear on s3).
>
> any ideas how to debug this?
>
> see the thread dump below:
>
> "Attach Listener" daemon prio=10 tid=0x00007f67a8001000 nid=0x7b90 waiting
> on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "SparkUI-224" daemon prio=10 tid=0x00007f6778001000 nid=0x7b4a waiting on
> condition [0x00007f67c2cfa000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c77cc010> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>         at
>
> org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
>         at java.lang.Thread.run(Thread.java:745)
>
> "SparkUI-223" daemon prio=10 tid=0x00007f671008e000 nid=0x7b49 waiting on
> condition [0x00007f67d1b4d000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c77cc010> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>         at
>
> org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
>         at java.lang.Thread.run(Thread.java:745)
>
> "SparkUI-222" daemon prio=10 tid=0x00007f677c006800 nid=0x7b48 waiting on
> condition [0x00007f67c8e33000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c77cc010> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>         at
>
> org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
>         at java.lang.Thread.run(Thread.java:745)
>
> "DestroyJavaVM" prio=10 tid=0x00007f67ed157000 nid=0x23c9 waiting on
> condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "SparkUI-205" daemon prio=10 tid=0x00007f6718002000 nid=0x7aa0 waiting on
> condition [0x00007f67b82ee000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c77cc010> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>         at
>
> org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
>         at java.lang.Thread.run(Thread.java:745)
>
> "Scheduler-916842649" prio=10 tid=0x00007f672c004000 nid=0x7a9e waiting on
> condition [0x00007f67b89f5000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c78ee520> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-7" daemon prio=10 tid=0x00007f66e4007800 nid=0x61cf
> runnable
> [0x00007f67c8429000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c7bdd8e0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c7bdd940> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c7bdd838> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>         at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-3" daemon prio=10 tid=0x00007f676c009000 nid=0x24b6
> waiting on condition [0x00007f67b8bf7000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c78ef7d0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-2" daemon prio=10 tid=0x00007f6764007000 nid=0x24b5
> waiting on condition [0x00007f67b8cf8000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c78ef7d0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-1" daemon prio=10 tid=0x00007f675c004000 nid=0x24b4
> waiting on condition [0x00007f67b8df9000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c78ef7d0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "task-result-getter-0" daemon prio=10 tid=0x00007f6768010000 nid=0x24b3
> waiting on condition [0x00007f67c862b000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c78ef7d0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-6" daemon prio=10 tid=0x00007f66e4011800 nid=0x24b2
> runnable
> [0x00007f66cbef9000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c8de9f60> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c8de9f50> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c8de9f80> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>         at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-5" daemon prio=10 tid=0x00007f66e4010000 nid=0x24b1
> runnable
> [0x00007f67b8efa000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c7bde078> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c7bde0d8> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c7bddfe0> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>         at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-4" daemon prio=10 tid=0x00007f66e400c800 nid=0x24b0
> runnable
> [0x00007f66cdffc000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c8e06ae8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c8e06ad8> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c8e06b08> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>         at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-3" daemon prio=10 tid=0x00007f66e400b000 nid=0x24af
> runnable
> [0x00007f67b8ffb000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c8e0d190> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c8e0d180> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c8e0d1b0> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>         at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-2" daemon prio=10 tid=0x00007f66e4008800 nid=0x24ae
> runnable
> [0x00007f67c01e2000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c8e49100> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c8e490f0> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c8e49120> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>         at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" daemon prio=10 tid=0x00007f66e400a000 nid=0x24ad
> runnable
> [0x00007f67d1f8e000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c8d31698> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c8d316f8> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c8d31600> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>         at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
>
> "java-sdk-http-connection-reaper" daemon prio=10 tid=0x000000000256f000
> nid=0x24ac waiting on condition [0x00007f67c04f0000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at
> com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:112)
>
> "shuffle-server-7" daemon prio=10 tid=0x00007f674800d000 nid=0x24ab
> runnable
> [0x00007f67c25f3000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c6603918> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c66049d8> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c6603870> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>         at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-6" daemon prio=10 tid=0x00007f674800b000 nid=0x24aa
> runnable
> [0x00007f67c26f4000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c78aa328> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c78aa348> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c78aa2e0> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>         at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-5" daemon prio=10 tid=0x00007f6748009000 nid=0x24a9
> runnable
> [0x00007f67c27f5000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c8ccfca0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c8cd0d70> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c8ccfbf8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>         at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-4" daemon prio=10 tid=0x00007f6748007000 nid=0x24a8
> runnable
> [0x00007f67c28f6000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c54e4cf0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c54e4db0> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c54e4c48> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>         at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-3" daemon prio=10 tid=0x00007f6748005000 nid=0x24a7
> runnable
> [0x00007f67c29f7000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c65dae00> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c65de9d8> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c65dad58> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>         at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-2" daemon prio=10 tid=0x00007f6748003800 nid=0x24a6
> runnable
> [0x00007f67c2af8000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c8ccfa18> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c77ce728> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c8ccf970> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>         at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-1" daemon prio=10 tid=0x00007f6748001800 nid=0x24a5
> runnable
> [0x00007f67c2bf9000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c54bf8c0> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c54c3fb0> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c54bf818> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>         at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
>
> "SparkListenerBus" daemon prio=10 tid=0x00007f6770e64800 nid=0x24a2 waiting
> on condition [0x00007f67c2dfb000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c7a97800> (a
> java.util.concurrent.Semaphore$NonfairSync)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>         at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
>         at
>
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiveListenerBus.scala:67)
>         at
>
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:66)
>         at
>
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:66)
>         at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
>         at
>
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:65)
>         at
> org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1229)
>         at
>
> org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:64)
>
> "context-cleaner-periodic-gc" daemon prio=10 tid=0x00007f6770e61000
> nid=0x24a1 waiting on condition [0x00007f67c2efc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c78f33e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "Spark Context Cleaner" daemon prio=10 tid=0x00007f6770e5f000 nid=0x24a0 in
> Object.wait() [0x00007f67c2ffd000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00000000c78caac0> (a
> java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>         - locked <0x00000000c78caac0> (a java.lang.ref.ReferenceQueue$Lock)
>         at
>
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:175)
>         at
> org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1229)
>         at
> org.apache.spark.ContextCleaner.org
> $apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:172)
>         at
> org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:67)
>
> "shuffle-server-0" daemon prio=10 tid=0x00007f6770bf9800 nid=0x249f
> runnable
> [0x00007f67c8126000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c8d55b88> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c8d5ce98> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c8d55af0> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>         at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
>
> "threadDeathWatcher-2-1" daemon prio=10 tid=0x00007f66ec00f800 nid=0x249e
> waiting on condition [0x00007f67c8227000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>         at
>
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>         at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-client-0" daemon prio=10 tid=0x00007f66e8003800 nid=0x249d
> runnable
> [0x00007f67c8328000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c31e90a8> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c31e90c8> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c31e9060> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>         at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
>
> "appclient-registration-retry-thread" daemon prio=10 tid=0x00007f6760003800
> nid=0x249b waiting on condition [0x00007f67c852a000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c3207ec0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "driver-revive-thread" daemon prio=10 tid=0x00007f676800d000 nid=0x2499
> waiting on condition [0x00007f67c872c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c8d90dd0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "dag-scheduler-event-loop" daemon prio=10 tid=0x00007f6770be7000 nid=0x2498
> waiting on condition [0x00007f67c882d000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c8d897c8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
>
> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:489)
>         at
> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:678)
>         at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
>
> "netty-rpc-env-timeout" daemon prio=10 tid=0x00007f670c00c800 nid=0x2497
> waiting on condition [0x00007f67c892e000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c32087c0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "Timer-0" daemon prio=10 tid=0x00007f6770bb1800 nid=0x2496 in Object.wait()
> [0x00007f67c8a2f000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00000000c8d884f8> (a java.util.TaskQueue)
>         at java.lang.Object.wait(Object.java:503)
>         at java.util.TimerThread.mainLoop(Timer.java:526)
>         - locked <0x00000000c8d884f8> (a java.util.TaskQueue)
>         at java.util.TimerThread.run(Timer.java:505)
>
> "heartbeat-receiver-event-loop-thread" daemon prio=10
> tid=0x00007f675c001800
> nid=0x2495 waiting on condition [0x00007f67c8b30000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c78ef1f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "SparkUI-59-acceptor-0@26f08623-ServerConnector@78f677cd{HTTP/1.1}{
> 0.0.0.0:4040}"
> daemon prio=10 tid=0x00007f6770ad9800 nid=0x2403 runnable
> [0x00007f67c8f34000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>         at
> sun.nio.ch
> .ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>         - locked <0x00000000c77cd9f8> (a java.lang.Object)
>         at
>
> org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:377)
>         at
>
> org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>         at java.lang.Thread.run(Thread.java:745)
>
> "SparkUI-58-selector-ServerConnectorManager@37faa18/3" daemon prio=10
> tid=0x00007f6770ad7800 nid=0x2402 runnable [0x00007f67c9035000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c77ccf98> (a sun.nio.ch.Util$2)
>         - locked <0x00000000c77ccf88> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c77cce70> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>         at
> org.spark_project.jetty.io
> .SelectorManager$ManagedSelector.select(SelectorManager.java:601)
>         at
> org.spark_project.jetty.io
> .SelectorManager$ManagedSelector.run(SelectorManager.java:550)
>         at
>
> org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>         at java.lang.Thread.run(Thread.java:745)
>
> "SparkUI-57-selector-ServerConnectorManager@37faa18/2" daemon prio=10
> tid=0x00007f6770ad6000 nid=0x2401 runnable [0x00007f67c9136000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c77cc4b0> (a sun.nio.ch.Util$2)
>         - locked <0x00000000c77cc4a0> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c77cc388> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>         at
> org.spark_project.jetty.io
> .SelectorManager$ManagedSelector.select(SelectorManager.java:601)
>         at
> org.spark_project.jetty.io
> .SelectorManager$ManagedSelector.run(SelectorManager.java:550)
>         at
>
> org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>         at java.lang.Thread.run(Thread.java:745)
>
> "SparkUI-56-selector-ServerConnectorManager@37faa18/1" daemon prio=10
> tid=0x00007f6770ad2000 nid=0x2400 runnable [0x00007f67c9237000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c77cd518> (a sun.nio.ch.Util$2)
>         - locked <0x00000000c77cd508> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c77cd3f0> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>         at
> org.spark_project.jetty.io
> .SelectorManager$ManagedSelector.select(SelectorManager.java:601)
>         at
> org.spark_project.jetty.io
> .SelectorManager$ManagedSelector.run(SelectorManager.java:550)
>         at
>
> org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>         at java.lang.Thread.run(Thread.java:745)
>
> "SparkUI-55-selector-ServerConnectorManager@37faa18/0" daemon prio=10
> tid=0x00007f6770acc800 nid=0x23ff runnable [0x00007f67d01fb000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c77cca18> (a sun.nio.ch.Util$2)
>         - locked <0x00000000c77cca08> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c77cc8f0> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>         at
> org.spark_project.jetty.io
> .SelectorManager$ManagedSelector.select(SelectorManager.java:601)
>         at
> org.spark_project.jetty.io
> .SelectorManager$ManagedSelector.run(SelectorManager.java:550)
>         at
>
> org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>         at
>
> org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>         at java.lang.Thread.run(Thread.java:745)
>
> "map-output-dispatcher-7" daemon prio=10 tid=0x00007f67707b2800 nid=0x23fd
> waiting on condition [0x00007f67d02fc000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c8d39b30> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
>
> org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "map-output-dispatcher-6" daemon prio=10 tid=0x00007f67707b0800 nid=0x23fc
> waiting on condition [0x00007f67d03fd000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c8d39b30> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
>
> org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "map-output-dispatcher-5" daemon prio=10 tid=0x00007f67707ae800 nid=0x23fb
> waiting on condition [0x00007f67d04fe000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c8d39b30> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
>
> org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "map-output-dispatcher-4" daemon prio=10 tid=0x00007f67707ad000 nid=0x23fa
> waiting on condition [0x00007f67d05ff000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c8d39b30> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
>
> org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "map-output-dispatcher-3" daemon prio=10 tid=0x00007f67707ab000 nid=0x23f9
> waiting on condition [0x00007f67d0700000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c8d39b30> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
>
> org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "map-output-dispatcher-2" daemon prio=10 tid=0x00007f67707a9000 nid=0x23f8
> waiting on condition [0x00007f67d0801000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c8d39b30> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
>
> org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "map-output-dispatcher-1" daemon prio=10 tid=0x00007f67707a7800 nid=0x23f7
> waiting on condition [0x00007f67d0902000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c8d39b30> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
>
> org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "map-output-dispatcher-0" daemon prio=10 tid=0x00007f67707a6000 nid=0x23f6
> waiting on condition [0x00007f67d0a03000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c8d39b30> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
>
> org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "shuffle-server-0" daemon prio=10 tid=0x00007f677078d000 nid=0x23f5
> runnable
> [0x00007f67d0b04000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000c7c80d50> (a
> io.netty.channel.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c7c80e30> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c7c80ca8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>         at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
>
> "dispatcher-event-loop-7" daemon prio=10 tid=0x00007f677052c000 nid=0x23f3
> waiting on condition [0x00007f67d0c05000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c7b0d7d8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
> org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:205)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "dispatcher-event-loop-6" daemon prio=10 tid=0x00007f677052a000 nid=0x23f2
> waiting on condition [0x00007f67d0d06000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c7b0d7d8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
> org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:205)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "dispatcher-event-loop-5" daemon prio=10 tid=0x00007f6770528000 nid=0x23f1
> waiting on condition [0x00007f67d0e07000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c7b0d7d8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
> org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:205)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "dispatcher-event-loop-4" daemon prio=10 tid=0x00007f677051d800 nid=0x23f0
> waiting on condition [0x00007f67d0f08000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c7b0d7d8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
> org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:205)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "dispatcher-event-loop-3" daemon prio=10 tid=0x00007f677051c000 nid=0x23ef
> waiting on condition [0x00007f67d1009000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c7b0d7d8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
> org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:205)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "dispatcher-event-loop-2" daemon prio=10 tid=0x00007f677051a800 nid=0x23ee
> waiting on condition [0x00007f67d110a000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c7b0d7d8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
> org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:205)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "dispatcher-event-loop-1" daemon prio=10 tid=0x00007f6770523000 nid=0x23ed
> waiting on condition [0x00007f67d120b000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c7b0d7d8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
> org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:205)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "dispatcher-event-loop-0" daemon prio=10 tid=0x00007f6770522000 nid=0x23ec
> waiting on condition [0x00007f67d150d000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c7b0d7d8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
> org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:205)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" daemon prio=10 tid=0x00007f67ec0bd000 nid=0x23d8 runnable
> [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" daemon prio=10 tid=0x00007f67ec0ba800 nid=0x23d7
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" daemon prio=10 tid=0x00007f67ec0b7800 nid=0x23d6
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x00007f67ec0b5800 nid=0x23d5
> runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x00007f67ec088800 nid=0x23d4 in
> Object.wait() [0x00007f67d39f8000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00000000c7863020> (a
> java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>         - locked <0x00000000c7863020> (a java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
>         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" daemon prio=10 tid=0x00007f67ec086800 nid=0x23d3 in
> Object.wait() [0x00007f67d3af9000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00000000c7862b20> (a java.lang.ref.Reference$Lock)
>         at java.lang.Object.wait(Object.java:503)
>         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
>         - locked <0x00000000c7862b20> (a java.lang.ref.Reference$Lock)
>
> "VM Thread" prio=10 tid=0x00007f67ec082000 nid=0x23d2 runnable
>
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f67ec023000 nid=0x23ca
> runnable
>
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f67ec025000 nid=0x23cb
> runnable
>
> "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f67ec027000 nid=0x23cc
> runnable
>
> "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f67ec029000 nid=0x23cd
> runnable
>
> "GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f67ec02a800 nid=0x23ce
> runnable
>
> "GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f67ec02c800 nid=0x23cf
> runnable
>
> "GC task thread#6 (ParallelGC)" prio=10 tid=0x00007f67ec02e800 nid=0x23d0
> runnable
>
> "GC task thread#7 (ParallelGC)" prio=10 tid=0x00007f67ec030800 nid=0x23d1
> runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x00007f67ec0c8000 nid=0x23d9 waiting
> on condition
>
> JNI global references: 391
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-2-0-0-job-completes-but-hangs-tp19374.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Mime
View raw message