spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamas Jambor <jambo...@gmail.com>
Subject Re: Spark 2.0.0 job completes but hangs
Date Mon, 10 Oct 2016 16:35:49 GMT
Thanks Sean, that explains it.

On Mon, Oct 10, 2016 at 4:54 PM, Sean Owen <sowen@cloudera.com> wrote:

> I am pretty sure that's
> https://issues.apache.org/jira/browse/SPARK-17707
>
> On Mon, Oct 10, 2016, 16:26 jamborta <jamborta@gmail.com> wrote:
>
>> Hi all,
>>
>> I have a spark job that takes about an hour to run, in the end it
>> completes
>> all the task, then the job just hangs and does nothing (it writes to s3 as
>> the last step, which also gets completed, all files appear on s3).
>>
>> any ideas how to debug this?
>>
>> see the thread dump below:
>>
>> "Attach Listener" daemon prio=10 tid=0x00007f67a8001000 nid=0x7b90 waiting
>> on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "SparkUI-224" daemon prio=10 tid=0x00007f6778001000 nid=0x7b4a waiting on
>> condition [0x00007f67c2cfa000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c77cc010> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>>         at
>> org.spark_project.jetty.util.BlockingArrayQueue.poll(
>> BlockingArrayQueue.java:389)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool.
>> idleJobPoll(QueuedThreadPool.java:531)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool.
>> access$700(QueuedThreadPool.java:47)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(
>> QueuedThreadPool.java:590)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "SparkUI-223" daemon prio=10 tid=0x00007f671008e000 nid=0x7b49 waiting on
>> condition [0x00007f67d1b4d000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c77cc010> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>>         at
>> org.spark_project.jetty.util.BlockingArrayQueue.poll(
>> BlockingArrayQueue.java:389)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool.
>> idleJobPoll(QueuedThreadPool.java:531)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool.
>> access$700(QueuedThreadPool.java:47)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(
>> QueuedThreadPool.java:590)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "SparkUI-222" daemon prio=10 tid=0x00007f677c006800 nid=0x7b48 waiting on
>> condition [0x00007f67c8e33000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c77cc010> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>>         at
>> org.spark_project.jetty.util.BlockingArrayQueue.poll(
>> BlockingArrayQueue.java:389)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool.
>> idleJobPoll(QueuedThreadPool.java:531)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool.
>> access$700(QueuedThreadPool.java:47)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(
>> QueuedThreadPool.java:590)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "DestroyJavaVM" prio=10 tid=0x00007f67ed157000 nid=0x23c9 waiting on
>> condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "SparkUI-205" daemon prio=10 tid=0x00007f6718002000 nid=0x7aa0 waiting on
>> condition [0x00007f67b82ee000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c77cc010> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>>         at
>> org.spark_project.jetty.util.BlockingArrayQueue.poll(
>> BlockingArrayQueue.java:389)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool.
>> idleJobPoll(QueuedThreadPool.java:531)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool.
>> access$700(QueuedThreadPool.java:47)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(
>> QueuedThreadPool.java:590)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "Scheduler-916842649" prio=10 tid=0x00007f672c004000 nid=0x7a9e waiting on
>> condition [0x00007f67b89f5000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c78ee520> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(
>> ScheduledThreadPoolExecutor.java:1079)
>>         at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(
>> ScheduledThreadPoolExecutor.java:807)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.getTask(
>> ThreadPoolExecutor.java:1068)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1130)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-7" daemon prio=10 tid=0x00007f66e4007800 nid=0x61cf
>> runnable
>> [0x00007f67c8429000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c7bdd8e0> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>>         - locked <0x00000000c7bdd940> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c7bdd838> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at io.netty.channel.nio.NioEventLoop.select(
>> NioEventLoop.java:622)
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>         at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:111)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "task-result-getter-3" daemon prio=10 tid=0x00007f676c009000 nid=0x24b6
>> waiting on condition [0x00007f67b8bf7000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c78ef7d0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.getTask(
>> ThreadPoolExecutor.java:1068)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1130)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "task-result-getter-2" daemon prio=10 tid=0x00007f6764007000 nid=0x24b5
>> waiting on condition [0x00007f67b8cf8000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c78ef7d0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.getTask(
>> ThreadPoolExecutor.java:1068)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1130)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "task-result-getter-1" daemon prio=10 tid=0x00007f675c004000 nid=0x24b4
>> waiting on condition [0x00007f67b8df9000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c78ef7d0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.getTask(
>> ThreadPoolExecutor.java:1068)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1130)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "task-result-getter-0" daemon prio=10 tid=0x00007f6768010000 nid=0x24b3
>> waiting on condition [0x00007f67c862b000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c78ef7d0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.getTask(
>> ThreadPoolExecutor.java:1068)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1130)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-6" daemon prio=10 tid=0x00007f66e4011800 nid=0x24b2
>> runnable
>> [0x00007f66cbef9000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c8de9f60> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>>         - locked <0x00000000c8de9f50> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c8de9f80> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at io.netty.channel.nio.NioEventLoop.select(
>> NioEventLoop.java:622)
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>         at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:111)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-5" daemon prio=10 tid=0x00007f66e4010000 nid=0x24b1
>> runnable
>> [0x00007f67b8efa000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c7bde078> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>>         - locked <0x00000000c7bde0d8> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c7bddfe0> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at io.netty.channel.nio.NioEventLoop.select(
>> NioEventLoop.java:622)
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>         at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:111)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-4" daemon prio=10 tid=0x00007f66e400c800 nid=0x24b0
>> runnable
>> [0x00007f66cdffc000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c8e06ae8> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>>         - locked <0x00000000c8e06ad8> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c8e06b08> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at io.netty.channel.nio.NioEventLoop.select(
>> NioEventLoop.java:622)
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>         at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:111)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-3" daemon prio=10 tid=0x00007f66e400b000 nid=0x24af
>> runnable
>> [0x00007f67b8ffb000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c8e0d190> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>>         - locked <0x00000000c8e0d180> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c8e0d1b0> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at io.netty.channel.nio.NioEventLoop.select(
>> NioEventLoop.java:622)
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>         at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:111)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-2" daemon prio=10 tid=0x00007f66e4008800 nid=0x24ae
>> runnable
>> [0x00007f67c01e2000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c8e49100> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>>         - locked <0x00000000c8e490f0> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c8e49120> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at io.netty.channel.nio.NioEventLoop.select(
>> NioEventLoop.java:622)
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>         at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:111)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-1" daemon prio=10 tid=0x00007f66e400a000 nid=0x24ad
>> runnable
>> [0x00007f67d1f8e000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c8d31698> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>>         - locked <0x00000000c8d316f8> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c8d31600> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at io.netty.channel.nio.NioEventLoop.select(
>> NioEventLoop.java:622)
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>         at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:111)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "java-sdk-http-connection-reaper" daemon prio=10 tid=0x000000000256f000
>> nid=0x24ac waiting on condition [0x00007f67c04f0000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>         at java.lang.Thread.sleep(Native Method)
>>         at
>> com.amazonaws.http.IdleConnectionReaper.run(
>> IdleConnectionReaper.java:112)
>>
>> "shuffle-server-7" daemon prio=10 tid=0x00007f674800d000 nid=0x24ab
>> runnable
>> [0x00007f67c25f3000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c6603918> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>>         - locked <0x00000000c66049d8> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c6603870> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at io.netty.channel.nio.NioEventLoop.select(
>> NioEventLoop.java:622)
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>         at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:111)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-6" daemon prio=10 tid=0x00007f674800b000 nid=0x24aa
>> runnable
>> [0x00007f67c26f4000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c78aa328> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>>         - locked <0x00000000c78aa348> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c78aa2e0> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at io.netty.channel.nio.NioEventLoop.select(
>> NioEventLoop.java:622)
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>         at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:111)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-5" daemon prio=10 tid=0x00007f6748009000 nid=0x24a9
>> runnable
>> [0x00007f67c27f5000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c8ccfca0> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>>         - locked <0x00000000c8cd0d70> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c8ccfbf8> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at io.netty.channel.nio.NioEventLoop.select(
>> NioEventLoop.java:622)
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>         at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:111)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-4" daemon prio=10 tid=0x00007f6748007000 nid=0x24a8
>> runnable
>> [0x00007f67c28f6000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c54e4cf0> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>>         - locked <0x00000000c54e4db0> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c54e4c48> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at io.netty.channel.nio.NioEventLoop.select(
>> NioEventLoop.java:622)
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>         at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:111)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-3" daemon prio=10 tid=0x00007f6748005000 nid=0x24a7
>> runnable
>> [0x00007f67c29f7000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c65dae00> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>>         - locked <0x00000000c65de9d8> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c65dad58> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at io.netty.channel.nio.NioEventLoop.select(
>> NioEventLoop.java:622)
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>         at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:111)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-2" daemon prio=10 tid=0x00007f6748003800 nid=0x24a6
>> runnable
>> [0x00007f67c2af8000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c8ccfa18> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>>         - locked <0x00000000c77ce728> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c8ccf970> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at io.netty.channel.nio.NioEventLoop.select(
>> NioEventLoop.java:622)
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>         at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:111)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-1" daemon prio=10 tid=0x00007f6748001800 nid=0x24a5
>> runnable
>> [0x00007f67c2bf9000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c54bf8c0> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>>         - locked <0x00000000c54c3fb0> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c54bf818> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at io.netty.channel.nio.NioEventLoop.select(
>> NioEventLoop.java:622)
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>         at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:111)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "SparkListenerBus" daemon prio=10 tid=0x00007f6770e64800 nid=0x24a2
>> waiting
>> on condition [0x00007f67c2dfb000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c7a97800> (a
>> java.util.concurrent.Semaphore$NonfairSync)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.
>> parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.
>> doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.
>> acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>>         at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
>>         at
>> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$
>> anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(
>> LiveListenerBus.scala:67)
>>         at
>> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$
>> anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:66)
>>         at
>> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$
>> anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:66)
>>         at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
>>         at
>> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$
>> anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:65)
>>         at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.
>> scala:1229)
>>         at
>> org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(
>> LiveListenerBus.scala:64)
>>
>> "context-cleaner-periodic-gc" daemon prio=10 tid=0x00007f6770e61000
>> nid=0x24a1 waiting on condition [0x00007f67c2efc000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c78f33e8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>>         at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(
>> ScheduledThreadPoolExecutor.java:1090)
>>         at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(
>> ScheduledThreadPoolExecutor.java:807)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.getTask(
>> ThreadPoolExecutor.java:1068)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1130)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "Spark Context Cleaner" daemon prio=10 tid=0x00007f6770e5f000 nid=0x24a0
>> in
>> Object.wait() [0x00007f67c2ffd000]
>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>         at java.lang.Object.wait(Native Method)
>>         - waiting on <0x00000000c78caac0> (a java.lang.ref.ReferenceQueue$
>> Lock)
>>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>>         - locked <0x00000000c78caac0> (a java.lang.ref.ReferenceQueue$
>> Lock)
>>         at
>> org.apache.spark.ContextCleaner$$anonfun$org$
>> apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(
>> ContextCleaner.scala:175)
>>         at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.
>> scala:1229)
>>         at
>> org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$
>> keepCleaning(ContextCleaner.scala:172)
>>         at org.apache.spark.ContextCleaner$$anon$1.run(
>> ContextCleaner.scala:67)
>>
>> "shuffle-server-0" daemon prio=10 tid=0x00007f6770bf9800 nid=0x249f
>> runnable
>> [0x00007f67c8126000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c8d55b88> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>>         - locked <0x00000000c8d5ce98> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c8d55af0> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at io.netty.channel.nio.NioEventLoop.select(
>> NioEventLoop.java:622)
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>         at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:111)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "threadDeathWatcher-2-1" daemon prio=10 tid=0x00007f66ec00f800 nid=0x249e
>> waiting on condition [0x00007f67c8227000]
>>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>>         at java.lang.Thread.sleep(Native Method)
>>         at
>> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>         at
>> io.netty.util.concurrent.DefaultThreadFactory$
>> DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-client-0" daemon prio=10 tid=0x00007f66e8003800 nid=0x249d
>> runnable
>> [0x00007f67c8328000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c31e90a8> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>>         - locked <0x00000000c31e90c8> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c31e9060> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at io.netty.channel.nio.NioEventLoop.select(
>> NioEventLoop.java:622)
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>         at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:111)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "appclient-registration-retry-thread" daemon prio=10
>> tid=0x00007f6760003800
>> nid=0x249b waiting on condition [0x00007f67c852a000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c3207ec0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(
>> ScheduledThreadPoolExecutor.java:1079)
>>         at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(
>> ScheduledThreadPoolExecutor.java:807)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.getTask(
>> ThreadPoolExecutor.java:1068)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1130)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "driver-revive-thread" daemon prio=10 tid=0x00007f676800d000 nid=0x2499
>> waiting on condition [0x00007f67c872c000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c8d90dd0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>>         at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(
>> ScheduledThreadPoolExecutor.java:1090)
>>         at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(
>> ScheduledThreadPoolExecutor.java:807)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.getTask(
>> ThreadPoolExecutor.java:1068)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1130)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "dag-scheduler-event-loop" daemon prio=10 tid=0x00007f6770be7000
>> nid=0x2498
>> waiting on condition [0x00007f67c882d000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c8d897c8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingDeque.takeFirst(
>> LinkedBlockingDeque.java:489)
>>         at
>> java.util.concurrent.LinkedBlockingDeque.take(
>> LinkedBlockingDeque.java:678)
>>         at org.apache.spark.util.EventLoop$$anon$1.run(
>> EventLoop.scala:46)
>>
>> "netty-rpc-env-timeout" daemon prio=10 tid=0x00007f670c00c800 nid=0x2497
>> waiting on condition [0x00007f67c892e000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c32087c0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>>         at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(
>> ScheduledThreadPoolExecutor.java:1090)
>>         at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(
>> ScheduledThreadPoolExecutor.java:807)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.getTask(
>> ThreadPoolExecutor.java:1068)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1130)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "Timer-0" daemon prio=10 tid=0x00007f6770bb1800 nid=0x2496 in
>> Object.wait()
>> [0x00007f67c8a2f000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>>         at java.lang.Object.wait(Native Method)
>>         - waiting on <0x00000000c8d884f8> (a java.util.TaskQueue)
>>         at java.lang.Object.wait(Object.java:503)
>>         at java.util.TimerThread.mainLoop(Timer.java:526)
>>         - locked <0x00000000c8d884f8> (a java.util.TaskQueue)
>>         at java.util.TimerThread.run(Timer.java:505)
>>
>> "heartbeat-receiver-event-loop-thread" daemon prio=10
>> tid=0x00007f675c001800
>> nid=0x2495 waiting on condition [0x00007f67c8b30000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c78ef1f0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:226)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>>         at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(
>> ScheduledThreadPoolExecutor.java:1090)
>>         at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(
>> ScheduledThreadPoolExecutor.java:807)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.getTask(
>> ThreadPoolExecutor.java:1068)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1130)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "SparkUI-59-acceptor-0@26f08623-ServerConnector@78f677cd{HTTP/1.1}{
>> 0.0.0.0:4040}"
>> daemon prio=10 tid=0x00007f6770ad9800 nid=0x2403 runnable
>> [0x00007f67c8f34000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>         at
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:
>> 250)
>>         - locked <0x00000000c77cd9f8> (a java.lang.Object)
>>         at
>> org.spark_project.jetty.server.ServerConnector.accept(
>> ServerConnector.java:377)
>>         at
>> org.spark_project.jetty.server.AbstractConnector$Acceptor.run(
>> AbstractConnector.java:500)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool.
>> runJob(QueuedThreadPool.java:635)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(
>> QueuedThreadPool.java:555)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "SparkUI-58-selector-ServerConnectorManager@37faa18/3" daemon prio=10
>> tid=0x00007f6770ad7800 nid=0x2402 runnable [0x00007f67c9035000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c77ccf98> (a sun.nio.ch.Util$2)
>>         - locked <0x00000000c77ccf88> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c77cce70> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>         at
>> org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(
>> SelectorManager.java:601)
>>         at
>> org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(
>> SelectorManager.java:550)
>>         at
>> org.spark_project.jetty.util.thread.NonBlockingThread.run(
>> NonBlockingThread.java:52)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool.
>> runJob(QueuedThreadPool.java:635)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(
>> QueuedThreadPool.java:555)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "SparkUI-57-selector-ServerConnectorManager@37faa18/2" daemon prio=10
>> tid=0x00007f6770ad6000 nid=0x2401 runnable [0x00007f67c9136000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c77cc4b0> (a sun.nio.ch.Util$2)
>>         - locked <0x00000000c77cc4a0> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c77cc388> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>         at
>> org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(
>> SelectorManager.java:601)
>>         at
>> org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(
>> SelectorManager.java:550)
>>         at
>> org.spark_project.jetty.util.thread.NonBlockingThread.run(
>> NonBlockingThread.java:52)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool.
>> runJob(QueuedThreadPool.java:635)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(
>> QueuedThreadPool.java:555)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "SparkUI-56-selector-ServerConnectorManager@37faa18/1" daemon prio=10
>> tid=0x00007f6770ad2000 nid=0x2400 runnable [0x00007f67c9237000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c77cd518> (a sun.nio.ch.Util$2)
>>         - locked <0x00000000c77cd508> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c77cd3f0> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>         at
>> org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(
>> SelectorManager.java:601)
>>         at
>> org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(
>> SelectorManager.java:550)
>>         at
>> org.spark_project.jetty.util.thread.NonBlockingThread.run(
>> NonBlockingThread.java:52)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool.
>> runJob(QueuedThreadPool.java:635)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(
>> QueuedThreadPool.java:555)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "SparkUI-55-selector-ServerConnectorManager@37faa18/0" daemon prio=10
>> tid=0x00007f6770acc800 nid=0x23ff runnable [0x00007f67d01fb000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c77cca18> (a sun.nio.ch.Util$2)
>>         - locked <0x00000000c77cca08> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c77cc8f0> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>         at
>> org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(
>> SelectorManager.java:601)
>>         at
>> org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(
>> SelectorManager.java:550)
>>         at
>> org.spark_project.jetty.util.thread.NonBlockingThread.run(
>> NonBlockingThread.java:52)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool.
>> runJob(QueuedThreadPool.java:635)
>>         at
>> org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(
>> QueuedThreadPool.java:555)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "map-output-dispatcher-7" daemon prio=10 tid=0x00007f67707b2800 nid=0x23fd
>> waiting on condition [0x00007f67d02fc000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c8d39b30> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(
>> MapOutputTracker.scala:338)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "map-output-dispatcher-6" daemon prio=10 tid=0x00007f67707b0800 nid=0x23fc
>> waiting on condition [0x00007f67d03fd000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c8d39b30> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(
>> MapOutputTracker.scala:338)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "map-output-dispatcher-5" daemon prio=10 tid=0x00007f67707ae800 nid=0x23fb
>> waiting on condition [0x00007f67d04fe000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c8d39b30> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(
>> MapOutputTracker.scala:338)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "map-output-dispatcher-4" daemon prio=10 tid=0x00007f67707ad000 nid=0x23fa
>> waiting on condition [0x00007f67d05ff000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c8d39b30> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(
>> MapOutputTracker.scala:338)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "map-output-dispatcher-3" daemon prio=10 tid=0x00007f67707ab000 nid=0x23f9
>> waiting on condition [0x00007f67d0700000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c8d39b30> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(
>> MapOutputTracker.scala:338)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "map-output-dispatcher-2" daemon prio=10 tid=0x00007f67707a9000 nid=0x23f8
>> waiting on condition [0x00007f67d0801000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c8d39b30> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(
>> MapOutputTracker.scala:338)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "map-output-dispatcher-1" daemon prio=10 tid=0x00007f67707a7800 nid=0x23f7
>> waiting on condition [0x00007f67d0902000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c8d39b30> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(
>> MapOutputTracker.scala:338)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "map-output-dispatcher-0" daemon prio=10 tid=0x00007f67707a6000 nid=0x23f6
>> waiting on condition [0x00007f67d0a03000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c8d39b30> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(
>> MapOutputTracker.scala:338)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "shuffle-server-0" daemon prio=10 tid=0x00007f677078d000 nid=0x23f5
>> runnable
>> [0x00007f67d0b04000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.
>> java:79)
>>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>         - locked <0x00000000c7c80d50> (a
>> io.netty.channel.nio.SelectedSelectionKeySet)
>>         - locked <0x00000000c7c80e30> (a java.util.Collections$
>> UnmodifiableSet)
>>         - locked <0x00000000c7c80ca8> (a sun.nio.ch.EPollSelectorImpl)
>>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>         at io.netty.channel.nio.NioEventLoop.select(
>> NioEventLoop.java:622)
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>         at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.
>> run(SingleThreadEventExecutor.java:111)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "dispatcher-event-loop-7" daemon prio=10 tid=0x00007f677052c000 nid=0x23f3
>> waiting on condition [0x00007f67d0c05000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c7b0d7d8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(
>> Dispatcher.scala:205)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "dispatcher-event-loop-6" daemon prio=10 tid=0x00007f677052a000 nid=0x23f2
>> waiting on condition [0x00007f67d0d06000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c7b0d7d8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(
>> Dispatcher.scala:205)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "dispatcher-event-loop-5" daemon prio=10 tid=0x00007f6770528000 nid=0x23f1
>> waiting on condition [0x00007f67d0e07000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c7b0d7d8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(
>> Dispatcher.scala:205)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "dispatcher-event-loop-4" daemon prio=10 tid=0x00007f677051d800 nid=0x23f0
>> waiting on condition [0x00007f67d0f08000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c7b0d7d8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(
>> Dispatcher.scala:205)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "dispatcher-event-loop-3" daemon prio=10 tid=0x00007f677051c000 nid=0x23ef
>> waiting on condition [0x00007f67d1009000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c7b0d7d8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(
>> Dispatcher.scala:205)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "dispatcher-event-loop-2" daemon prio=10 tid=0x00007f677051a800 nid=0x23ee
>> waiting on condition [0x00007f67d110a000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c7b0d7d8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(
>> Dispatcher.scala:205)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "dispatcher-event-loop-1" daemon prio=10 tid=0x00007f6770523000 nid=0x23ed
>> waiting on condition [0x00007f67d120b000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c7b0d7d8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(
>> Dispatcher.scala:205)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "dispatcher-event-loop-0" daemon prio=10 tid=0x00007f6770522000 nid=0x23ec
>> waiting on condition [0x00007f67d150d000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x00000000c7b0d7d8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>         at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:186)
>>         at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>>         at
>> java.util.concurrent.LinkedBlockingQueue.take(
>> LinkedBlockingQueue.java:442)
>>         at
>> org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(
>> Dispatcher.scala:205)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> "Service Thread" daemon prio=10 tid=0x00007f67ec0bd000 nid=0x23d8 runnable
>> [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "C2 CompilerThread1" daemon prio=10 tid=0x00007f67ec0ba800 nid=0x23d7
>> waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "C2 CompilerThread0" daemon prio=10 tid=0x00007f67ec0b7800 nid=0x23d6
>> waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Signal Dispatcher" daemon prio=10 tid=0x00007f67ec0b5800 nid=0x23d5
>> runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Finalizer" daemon prio=10 tid=0x00007f67ec088800 nid=0x23d4 in
>> Object.wait() [0x00007f67d39f8000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>>         at java.lang.Object.wait(Native Method)
>>         - waiting on <0x00000000c7863020> (a java.lang.ref.ReferenceQueue$
>> Lock)
>>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>>         - locked <0x00000000c7863020> (a java.lang.ref.ReferenceQueue$
>> Lock)
>>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
>>         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.
>> java:209)
>>
>> "Reference Handler" daemon prio=10 tid=0x00007f67ec086800 nid=0x23d3 in
>> Object.wait() [0x00007f67d3af9000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>>         at java.lang.Object.wait(Native Method)
>>         - waiting on <0x00000000c7862b20> (a java.lang.ref.Reference$Lock)
>>         at java.lang.Object.wait(Object.java:503)
>>         at java.lang.ref.Reference$ReferenceHandler.run(
>> Reference.java:133)
>>         - locked <0x00000000c7862b20> (a java.lang.ref.Reference$Lock)
>>
>> "VM Thread" prio=10 tid=0x00007f67ec082000 nid=0x23d2 runnable
>>
>> "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f67ec023000 nid=0x23ca
>> runnable
>>
>> "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f67ec025000 nid=0x23cb
>> runnable
>>
>> "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f67ec027000 nid=0x23cc
>> runnable
>>
>> "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f67ec029000 nid=0x23cd
>> runnable
>>
>> "GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f67ec02a800 nid=0x23ce
>> runnable
>>
>> "GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f67ec02c800 nid=0x23cf
>> runnable
>>
>> "GC task thread#6 (ParallelGC)" prio=10 tid=0x00007f67ec02e800 nid=0x23d0
>> runnable
>>
>> "GC task thread#7 (ParallelGC)" prio=10 tid=0x00007f67ec030800 nid=0x23d1
>> runnable
>>
>> "VM Periodic Task Thread" prio=10 tid=0x00007f67ec0c8000 nid=0x23d9
>> waiting
>> on condition
>>
>> JNI global references: 391
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://apache-spark-
>> developers-list.1001551.n3.nabble.com/Spark-2-0-0-job-
>> completes-but-hangs-tp19374.html
>> Sent from the Apache Spark Developers List mailing list archive at
>> Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>

Mime
View raw message