spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Perez <christ...@svds.com>
Subject Re: Spark SQL queries hang forever
Date Thu, 02 Apr 2015 19:32:23 GMT
I found that rewriting our tables in Impala (originally Hive) solved
the problem. The files are bigger so that there are 180 instead of
718.

I also think that turning the shuffle.partitions down to 100 on the
original set prevented Spark from hanging, but haven't tried to
reproduce since finding this workaround.

Perhaps there's an issue with either a) hidden Spark issues with the
particular parquet files or b) subtle bugs that appear later when
Spark processes many files. **A shuffle hang occurred with >800 text
files of same data.**

On Wed, Apr 1, 2015 at 6:39 AM, Marius Soutier <mps.dev@gmail.com> wrote:
> I have similiar problem since upgrading to 1.2.1.
>
> SparkSQL queries on small, simple Parquet files take up to 10 minutes. After an initial hang, it seems to work fine for a while.
>
> The problematic stage is RangePartitioner at Exchange.scala:88.
>
>
>> On 31.03.2015, at 21:55, Christian Perez <christian@svds.com> wrote:
>>
>> I think I've seen a timeout in one of the executor logs... but I'm
>> usually not that patient.
>>
>> If I kill the executor running the last task started on spark UI, 2
>> out of 3 times it will retry and complete successfully.
>>
>> I've found that everything works when running in local mode versus
>> YARN. Have not tried Spark standalone. Something (block fetcher?) is
>> hanging during shuffle when distributed...
>>
>> On Tue, Mar 31, 2015 at 12:10 AM, Zsolt Tóth <toth.zsolt.bme@gmail.com> wrote:
>>> I have a very similar problem with collect() but only on relatively large
>>> input data, on Amazon EMR. Look for my thread for details if interested.
>>> I get "connection timed out" exception after a 15-20 minutes in one of the
>>> container logs - does it happen in your case?
>>>
>>> 2015-03-30 23:34 GMT+02:00 Christian Perez <christian@svds.com>:
>>>>
>>>> I'm having the exact same problem (hanging) with takeOrdered (using
>>>> PySpark's orderBy on a DataFrame.)
>>>>
>>>> On Thu, Mar 26, 2015 at 7:21 PM, Jon Chase <jon.chase@gmail.com> wrote:
>>>>> Here are the jstacks from a 3 worker cluster.  Two of the workers are
>>>>> sitting with 4 active tasks, the other has no active tasks.  Not sure if
>>>>> it
>>>>> makes a difference, but I only see an ExecutorLauncher instance via JPS
>>>>> on
>>>>> the worker that has no active tasks.
>>>>>
>>>>> Here's the first stack dump, this machine has 4 active tasks:
>>>>>
>>>>> 2015-03-27 02:11:46
>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>>>>> mode):
>>>>>
>>>>> "Attach Listener" #74 daemon prio=9 os_prio=0 tid=0x00007f91ec12e000
>>>>> nid=0x43a3 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "sparkExecutor-akka.actor.default-dispatcher-14" #72 daemon prio=5
>>>>> os_prio=0
>>>>> tid=0x00007f91d02fd000 nid=0x42ee waiting on condition
>>>>> [0x00007f91bad9a000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000589884fc0> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "shuffle-server-2" #52 daemon prio=5 os_prio=0 tid=0x00007f91d8379000
>>>>> nid=0x42e2 runnable [0x00007f91b4c54000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000005898aa000> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x00000005898c4b80> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005898a6798> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-client-2" #48 daemon prio=5 os_prio=0 tid=0x00007f91d80ae800
>>>>> nid=0x42e1 runnable [0x00007f91b5d56000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058ad5b5f8> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058ad5b380> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x0000000589a27c78> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-server-1" #51 daemon prio=5 os_prio=0 tid=0x00007f91d80ac000
>>>>> nid=0x42e0 runnable [0x00007f91b5e57000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000005898aa0c0> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x00000005898dac48> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005898a6828> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-client-1" #47 daemon prio=5 os_prio=0 tid=0x00007f91d01f7800
>>>>> nid=0x42df runnable [0x00007f91b5f58000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058ad5b6b8> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058ad6f698> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058ad5e440> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91d0773800 nid=0x427b waiting on condition
>>>>> [0x00007f91b6258000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000006cfda6800> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>>> at
>>>>>
>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "org.apache.hadoop.hdfs.PeerCache@40a24c94" #67 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91d0c86000 nid=0x41f7 waiting on condition
>>>>> [0x00007f91b6b5a000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>>>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>>>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Thread-8" #65 daemon prio=5 os_prio=0 tid=0x00007f91ed497000 nid=0x41f2
>>>>> waiting on condition [0x00007f91b6c5b000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x000000058af8e0b8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Thread-5" #61 daemon prio=5 os_prio=0 tid=0x0000000000e1b000 nid=0x41ef
>>>>> runnable [0x00007f91baf9c000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "threadDeathWatcher-2-1" #60 daemon prio=1 os_prio=0
>>>>> tid=0x00007f91d8118800
>>>>> nid=0x41ee waiting on condition [0x00007f91bae9b000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-client-0" #46 daemon prio=5 os_prio=0 tid=0x0000000000961000
>>>>> nid=0x41ed runnable [0x00007f91baa97000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x0000000589224af0> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x00000005894b8d70> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x0000000589105ac0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Executor task launch worker-2" #58 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91d0019000 nid=0x41ea waiting on condition
>>>>> [0x00007f91bac98000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000006c8837750> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>>> at
>>>>>
>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Executor task launch worker-1" #57 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91d0016800 nid=0x41e9 waiting on condition
>>>>> [0x00007f91ba16d000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000006cc53bff0> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>>> at
>>>>>
>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Executor task launch worker-0" #56 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91d0015000 nid=0x41e8 waiting on condition
>>>>> [0x00007f91b9f6b000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000006cfda69b0> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>>> at
>>>>>
>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Driver Heartbeater" #55 daemon prio=5 os_prio=0 tid=0x000000000144e800
>>>>> nid=0x41c7 waiting on condition [0x00007f91bb9c6000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>>>>
>>>>> "shuffle-server-0" #50 daemon prio=5 os_prio=0 tid=0x0000000000a53800
>>>>> nid=0x41c6 runnable [0x00007f91bbac7000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058984ee60> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058984c368> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058984e688> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "metrics-ganglia-reporter-1-thread-1" #45 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91edcb7000 nid=0x41c3 waiting on condition
>>>>> [0x00007f91bbbc8000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005898f7290> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "BROADCAST_VARS cleanup timer" #44 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91eda6e000 nid=0x41c2 in Object.wait() [0x00007f91bbdca000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> - waiting on <0x000000058974cb00> (a java.util.TaskQueue)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x000000058974cb00> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "BLOCK_MANAGER cleanup timer" #43 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91ed9b3000 nid=0x41c1 in Object.wait() [0x00007f91bbcc9000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> - waiting on <0x0000000589904038> (a java.util.TaskQueue)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x0000000589904038> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "ForkJoinPool-3-worker-7" #40 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91ed92d800
>>>>> nid=0x41bf waiting on condition [0x00007f91bab98000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000589884e80> (a
>>>>> scala.concurrent.forkjoin.ForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "Hashed wheel timer #2" #35 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91ed92c800
>>>>> nid=0x41be waiting on condition [0x00007f91ba370000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O server boss #12" #39 daemon prio=5 os_prio=0
>>>>> tid=0x0000000000a02000
>>>>> nid=0x41bd runnable [0x00007f91b9564000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058992f188> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058992f170> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058992e3f0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #11" #38 daemon prio=5 os_prio=0 tid=0x0000000000a00000
>>>>> nid=0x41bc runnable [0x00007f91b9665000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x0000000589931218> (a sun.nio.ch.Util$2)
>>>>> - locked <0x0000000589931200> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058992e4c8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #10" #37 daemon prio=5 os_prio=0 tid=0x00000000009fe800
>>>>> nid=0x41bb runnable [0x00007f91b9766000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x0000000589944b50> (a sun.nio.ch.Util$2)
>>>>> - locked <0x0000000589944b38> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058992e558> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O boss #9" #36 daemon prio=5 os_prio=0 tid=0x0000000000d00800
>>>>> nid=0x41ba runnable [0x00007f91b9867000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x0000000589977fc8> (a sun.nio.ch.Util$2)
>>>>> - locked <0x0000000589977fb0> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058992e6c0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #8" #34 daemon prio=5 os_prio=0 tid=0x0000000000cfe800
>>>>> nid=0x41b9 runnable [0x00007f91b9968000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058998e4d8> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058998e4c0> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058992e798> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #7" #33 daemon prio=5 os_prio=0 tid=0x00000000009b5000
>>>>> nid=0x41b8 runnable [0x00007f91b9c69000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x0000000589990be0> (a sun.nio.ch.Util$2)
>>>>> - locked <0x0000000589990bc8> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005899a46a8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-6" #32 daemon
>>>>> prio=5
>>>>> os_prio=0 tid=0x00000000009b3800 nid=0x41b7 waiting on condition
>>>>> [0x00007f91b9d6a000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000589884f20> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-5" #31 daemon
>>>>> prio=5
>>>>> os_prio=0 tid=0x0000000000c0c800 nid=0x41b6 waiting on condition
>>>>> [0x00007f91b9e6b000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000589884f20> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
>>>>> tid=0x00007f91ed8f9800 nid=0x41b2 sleeping[0x00007f91ba26f000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>>>> at
>>>>>
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>>>> at
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f91ec126800
>>>>> nid=0x419e runnable [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f91ec109000
>>>>> nid=0x419d waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f91ec107000
>>>>> nid=0x419c waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f91ec105000
>>>>> nid=0x419b waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f91ec102800
>>>>> nid=0x419a runnable [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
>>>>> tid=0x00007f91ec101000 nid=0x4199 waiting on condition
>>>>> [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f91ec0cc000 nid=0x4198
>>>>> in
>>>>> Object.wait() [0x00007f91d4254000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>>> - locked <0x0000000589a3fbd8> (a java.lang.ref.ReferenceQueue$Lock)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>>>
>>>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f91ec0c9800
>>>>> nid=0x4197 in Object.wait() [0x00007f91d4355000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>>> - locked <0x0000000589a3fc20> (a java.lang.ref.Reference$Lock)
>>>>>
>>>>> "main" #1 prio=5 os_prio=0 tid=0x00007f91ec018800 nid=0x4190 waiting on
>>>>> condition [0x00007f91f1fdf000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000589a1b5f0> (a
>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>> at
>>>>>
>>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>>>> at
>>>>>
>>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>>>> at
>>>>>
>>>>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>>>> at scala.concurrent.Await$.ready(package.scala:86)
>>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>>>>
>>>>> "VM Thread" os_prio=0 tid=0x00007f91ec0c4800 nid=0x4196 runnable
>>>>>
>>>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02a000
>>>>> nid=0x4191 runnable
>>>>>
>>>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02b800
>>>>> nid=0x4192 runnable
>>>>>
>>>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02d800
>>>>> nid=0x4193 runnable
>>>>>
>>>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f91ec02f000
>>>>> nid=0x4194 runnable
>>>>>
>>>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f91ec072000
>>>>> nid=0x4195 runnable
>>>>>
>>>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f91ec139800 nid=0x419f
>>>>> waiting on condition
>>>>>
>>>>> JNI global references: 319
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Here's the next worker, also for active tasks:
>>>>>
>>>>> 2015-03-27 02:12:54
>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>>>>> mode):
>>>>>
>>>>> "Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007fcbfa3b6000
>>>>> nid=0x44f1 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "sparkExecutor-akka.actor.default-dispatcher-16" #76 daemon prio=5
>>>>> os_prio=0
>>>>> tid=0x00007fcbe488b000 nid=0x4464 waiting on condition
>>>>> [0x00007fcbbf87f000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000589510428> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "Executor task launch worker-8" #75 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbe4028800 nid=0x445f waiting on condition
>>>>> [0x00007fcbbf97f000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000006126cd358> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>>> at
>>>>>
>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0800
>>>>> nid=0x4459 runnable [0x00007fcbc88c7000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a6b8c98> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058a6bd270> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a6b97a8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007fcbfa0c0000
>>>>> nid=0x4458 runnable [0x00007fcbc7897000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a6e5a18> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058a6e4f60> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a6b9838> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007fcbf8113800
>>>>> nid=0x4457 runnable [0x00007fcbc7998000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058aade110> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058aadd508> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x0000000589657b38> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00007fcbf810e000
>>>>> nid=0x4456 runnable [0x00007fcbc7b9a000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058aade1d0> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058aaf16b8> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x0000000589657bc8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Executor task launch worker-7" #74 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbe48a5000 nid=0x4455 waiting on condition
>>>>> [0x00007fcbc4886000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000615539dd8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>>> at
>>>>>
>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Executor task launch worker-5" #72 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbe4165000 nid=0x4453 waiting on condition
>>>>> [0x00007fcbca620000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000617bd4238> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>>> at
>>>>>
>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Executor task launch worker-4" #71 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbe4164000 nid=0x4452 waiting on condition
>>>>> [0x00007fcbc87c5000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000006126cd538> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:278)
>>>>> at
>>>>>
>>>>> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
>>>>> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>>>>> at
>>>>>
>>>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:152)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:147)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:634)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at
>>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "org.apache.hadoop.hdfs.PeerCache@34fa4752" #69 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbdcff1800 nid=0x436e waiting on condition
>>>>> [0x00007fcbc4485000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>>>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>>>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x000000000309c800 nid=0x4369
>>>>> waiting on condition [0x00007fcbc4586000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x000000058adabe70> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007fcbfa32f800 nid=0x4366
>>>>> runnable [0x00007fcbc7e9d000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
>>>>> tid=0x00007fcbf9608800
>>>>> nid=0x4365 waiting on condition [0x00007fcbc89c8000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x000000000224b800
>>>>> nid=0x4364 runnable [0x00007fcbc8ac9000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x0000000589116ae0> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x00000005892ef140> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x0000000589171928> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x0000000003ecc000
>>>>> nid=0x4341 waiting on condition [0x00007fcbc95f4000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>>>>
>>>>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x0000000002345000
>>>>> nid=0x4340 runnable [0x00007fcbc94f3000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058964c068> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058a691b28> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058963b100> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbf9ed4800 nid=0x433d waiting on condition
>>>>> [0x00007fcbc97f6000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x000000058a6fa058> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbf9b4a000 nid=0x433c in Object.wait() [0x00007fcbc98f7000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> - waiting on <0x000000058928c438> (a java.util.TaskQueue)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x000000058928c438> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbf9b49000 nid=0x433b in Object.wait() [0x00007fcbc96f5000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> - waiting on <0x000000058928c5a0> (a java.util.TaskQueue)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x000000058928c5a0> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbdc29d000
>>>>> nid=0x433a waiting on condition [0x00007fcbc85c4000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbe4474800
>>>>> nid=0x4338 waiting on condition [0x00007fcbc86c5000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x000000058a70a468> (a
>>>>> scala.concurrent.forkjoin.ForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbdc29a000
>>>>> nid=0x4337 runnable [0x00007fcbc6e8f000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a724d68> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058a724d50> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a6b9a30> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007fcbdc299000
>>>>> nid=0x4336 runnable [0x00007fcbc6f90000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a73ee18> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058a73ee00> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a6b9b08> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007fcbdc356800
>>>>> nid=0x4335 runnable [0x00007fcbc7091000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a73f1b0> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058a73f198> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a7517d8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007fcbdc296000
>>>>> nid=0x4334 runnable [0x00007fcbc7392000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a796550> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058a796538> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a796888> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007fcbdc298000
>>>>> nid=0x4333 runnable [0x00007fcbc7493000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a7be840> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058a7be828> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a796960> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007fcbdc257000
>>>>> nid=0x4332 runnable [0x00007fcbc7594000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a7bed98> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058a7bed80> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a796a80> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon
>>>>> prio=5
>>>>> os_prio=0 tid=0x00000000022b2800 nid=0x4331 waiting on condition
>>>>> [0x00007fcbc7695000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005893c1820> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon
>>>>> prio=5
>>>>> os_prio=0 tid=0x00000000022b0800 nid=0x4330 waiting on condition
>>>>> [0x00007fcbc7796000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005893c1820> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
>>>>> tid=0x00007fcbf995d800 nid=0x432a waiting on condition
>>>>> [0x00007fcbc7d9c000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>>>> at
>>>>>
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>>>> at
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fcbf8118800
>>>>> nid=0x4318 runnable [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbf810b800
>>>>> nid=0x4317 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbf8109000
>>>>> nid=0x4316 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbf8107000
>>>>> nid=0x4315 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbf8105000
>>>>> nid=0x4314 runnable [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
>>>>> tid=0x00007fcbf8103800 nid=0x4313 waiting on condition
>>>>> [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbf80cc000 nid=0x4312
>>>>> in
>>>>> Object.wait() [0x00007fcbe1be7000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>>> - locked <0x000000058a7ed2d8> (a java.lang.ref.ReferenceQueue$Lock)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>>>
>>>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbf80c9800
>>>>> nid=0x4311 in Object.wait() [0x00007fcbe8150000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>>> - locked <0x000000058a7ed338> (a java.lang.ref.Reference$Lock)
>>>>>
>>>>> "main" #1 prio=5 os_prio=0 tid=0x00007fcbf8018800 nid=0x430a waiting on
>>>>> condition [0x00007fcbff9b9000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000589585398> (a
>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>> at
>>>>>
>>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>>>> at
>>>>>
>>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>>>> at
>>>>>
>>>>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>>>> at scala.concurrent.Await$.ready(package.scala:86)
>>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>>>>
>>>>> "VM Thread" os_prio=0 tid=0x00007fcbf80c4800 nid=0x4310 runnable
>>>>>
>>>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802a000
>>>>> nid=0x430b runnable
>>>>>
>>>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802b800
>>>>> nid=0x430c runnable
>>>>>
>>>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802d800
>>>>> nid=0x430d runnable
>>>>>
>>>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbf802f000
>>>>> nid=0x430e runnable
>>>>>
>>>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbf8072000
>>>>> nid=0x430f runnable
>>>>>
>>>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fcbf811b800 nid=0x4319
>>>>> waiting on condition
>>>>>
>>>>> JNI global references: 375
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> And the final worker, no active tasks:
>>>>>
>>>>>
>>>>> 2015-03-27 02:13:26
>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>>>>> mode):
>>>>>
>>>>> "Attach Listener" #81 daemon prio=9 os_prio=0 tid=0x00007f370db21000
>>>>> nid=0x3c5b waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "sparkExecutor-akka.actor.default-dispatcher-16" #77 daemon prio=5
>>>>> os_prio=0
>>>>> tid=0x00007f36f072d000 nid=0x3b9a waiting on condition
>>>>> [0x00007f36d2e33000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005895676e8> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "shuffle-server-2" #54 daemon prio=5 os_prio=0 tid=0x00007f370db23000
>>>>> nid=0x3b88 runnable [0x00007f36dafb9000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a86a7d8> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058a8818c8> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a867160> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-client-2" #50 daemon prio=5 os_prio=0 tid=0x00007f36f8bb9800
>>>>> nid=0x3b87 runnable [0x00007f36d7138000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058abea6c8> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058abea488> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x0000000589780fd8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-server-1" #53 daemon prio=5 os_prio=0 tid=0x00007f370d684800
>>>>> nid=0x3b86 runnable [0x00007f36daeb8000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a86a898> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058a89a400> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x000000058a8671f0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-client-1" #49 daemon prio=5 os_prio=0 tid=0x00000000015b1000
>>>>> nid=0x3b85 runnable [0x00007f36d7e3b000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058abea788> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058abff150> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x0000000589781068> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "org.apache.hadoop.hdfs.PeerCache@26764d41" #69 daemon prio=5 os_prio=0
>>>>> tid=0x00007f36f872d800 nid=0x3aa2 waiting on condition
>>>>> [0x00007f36d7a39000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>>>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>>>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Thread-8" #67 daemon prio=5 os_prio=0 tid=0x00000000018c8000 nid=0x3a9d
>>>>> waiting on condition [0x00007f36d7b3a000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x000000058aecd008> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Thread-5" #63 daemon prio=5 os_prio=0 tid=0x00007f36f04f4800 nid=0x3a9a
>>>>> runnable [0x00007f36db4be000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "threadDeathWatcher-2-1" #62 daemon prio=1 os_prio=0
>>>>> tid=0x00007f36f8025000
>>>>> nid=0x3a97 waiting on condition [0x00007f36dbfe9000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-client-0" #48 daemon prio=5 os_prio=0 tid=0x00007f36f8238000
>>>>> nid=0x3a96 runnable [0x00007f36dc0ea000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x0000000589202c58> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x0000000589390308> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005890f64f8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Driver Heartbeater" #57 daemon prio=5 os_prio=0 tid=0x00007f36f8586000
>>>>> nid=0x3a6d waiting on condition [0x00007f36dcc15000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:437)
>>>>>
>>>>> "shuffle-server-0" #52 daemon prio=5 os_prio=0 tid=0x00007f36f8ef9000
>>>>> nid=0x3a6c runnable [0x00007f36dcb14000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058a825c18> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x000000058a827090> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005895eb8d0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "metrics-ganglia-reporter-1-thread-1" #47 daemon prio=5 os_prio=0
>>>>> tid=0x00007f370dcb4000 nid=0x3a69 waiting on condition
>>>>> [0x00007f36dce17000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005897334d0> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "BROADCAST_VARS cleanup timer" #46 daemon prio=5 os_prio=0
>>>>> tid=0x00007f370da11800 nid=0x3a68 in Object.wait() [0x00007f36dcd16000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> - waiting on <0x00000005896b0d20> (a java.util.TaskQueue)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x00000005896b0d20> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "BLOCK_MANAGER cleanup timer" #45 daemon prio=5 os_prio=0
>>>>> tid=0x00007f370da10800 nid=0x3a67 in Object.wait() [0x00007f36dcf18000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> - waiting on <0x00000005896b0dc8> (a java.util.TaskQueue)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x00000005896b0dc8> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "Hashed wheel timer #2" #37 daemon prio=5 os_prio=0
>>>>> tid=0x00007f36f03a1000
>>>>> nid=0x3a66 waiting on condition [0x00007f36dbbe5000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "ForkJoinPool-3-worker-7" #42 daemon prio=5 os_prio=0
>>>>> tid=0x00007f370d9b2000
>>>>> nid=0x3a64 waiting on condition [0x00007f36dbde7000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005895675a8> (a
>>>>> scala.concurrent.forkjoin.ForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "New I/O server boss #12" #41 daemon prio=5 os_prio=0
>>>>> tid=0x00007f36f8713000
>>>>> nid=0x3a63 runnable [0x00007f36da4b0000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058970f538> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058970f520> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005896fbde0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #11" #40 daemon prio=5 os_prio=0 tid=0x00007f36f86e8800
>>>>> nid=0x3a62 runnable [0x00007f36da5b1000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000005896e86b0> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000005896e8698> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005896fbd08> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #10" #39 daemon prio=5 os_prio=0 tid=0x00007f36f86bc800
>>>>> nid=0x3a61 runnable [0x00007f36da6b2000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000005896e8578> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000005896e8560> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005891d3ef8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O boss #9" #38 daemon prio=5 os_prio=0 tid=0x00007f36f86a2000
>>>>> nid=0x3a60 runnable [0x00007f36da7b3000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x000000058969cee0> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000058969cec8> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005891d3be0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #8" #36 daemon prio=5 os_prio=0 tid=0x00007f36f8676800
>>>>> nid=0x3a5f runnable [0x00007f36da8b4000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000005896c8fe8> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000005896c8fd0> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005891d3dd8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #7" #35 daemon prio=5 os_prio=0 tid=0x00007f36f864c800
>>>>> nid=0x3a5e runnable [0x00007f36da9b5000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000005896b3340> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000005896b3328> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000005891d3cb8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-8" #34 daemon
>>>>> prio=5
>>>>> os_prio=0 tid=0x00007f36f862c000 nid=0x3a5d waiting on condition
>>>>> [0x00007f36dacb6000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000589567648> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkExecutor-akka.remote.default-remote-dispatcher-7" #33 daemon
>>>>> prio=5
>>>>> os_prio=0 tid=0x00007f36f862a000 nid=0x3a5c waiting on condition
>>>>> [0x00007f36dadb7000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000589567648> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkExecutor-scheduler-1" #27 daemon prio=5 os_prio=0
>>>>> tid=0x00007f370d97c000 nid=0x3a56 waiting on condition
>>>>> [0x00007f36db3bd000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>>>> at
>>>>>
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>>>> at
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f370c11e800
>>>>> nid=0x3a43 runnable [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f370c109000
>>>>> nid=0x3a42 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f370c107000
>>>>> nid=0x3a41 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f370c105000
>>>>> nid=0x3a40 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f370c102800
>>>>> nid=0x3a3f runnable [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
>>>>> tid=0x00007f370c101800 nid=0x3a3e waiting on condition
>>>>> [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f370c0cc000 nid=0x3a3d
>>>>> in
>>>>> Object.wait() [0x00007f36f51e5000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>>> - locked <0x000000058a99f0e0> (a java.lang.ref.ReferenceQueue$Lock)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>>>
>>>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f370c0c9800
>>>>> nid=0x3a3c in Object.wait() [0x00007f36f52e6000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>>> - locked <0x000000058a9a8b98> (a java.lang.ref.Reference$Lock)
>>>>>
>>>>> "main" #1 prio=5 os_prio=0 tid=0x00007f370c018800 nid=0x3a33 waiting on
>>>>> condition [0x00007f3712fda000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x000000058a974ab8> (a
>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>>>>> at
>>>>>
>>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
>>>>> at
>>>>>
>>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
>>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>>>> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
>>>>> at
>>>>>
>>>>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>>>> at scala.concurrent.Await$.ready(package.scala:86)
>>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
>>>>> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:174)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
>>>>> at
>>>>>
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>>>>
>>>>> "VM Thread" os_prio=0 tid=0x00007f370c0c4800 nid=0x3a3b runnable
>>>>>
>>>>> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02a000
>>>>> nid=0x3a34 runnable
>>>>>
>>>>> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02b800
>>>>> nid=0x3a35 runnable
>>>>>
>>>>> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02d800
>>>>> nid=0x3a36 runnable
>>>>>
>>>>> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f370c02f000
>>>>> nid=0x3a37 runnable
>>>>>
>>>>> "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f370c072000
>>>>> nid=0x3a3a runnable
>>>>>
>>>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007f370c121800 nid=0x3a44
>>>>> waiting on condition
>>>>>
>>>>> JNI global references: 486
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> And the driver:
>>>>>
>>>>> 2015-03-27 02:20:47
>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>>>>> mode):
>>>>>
>>>>> "qtp1576607881-152" #152 daemon prio=5 os_prio=0 tid=0x00007fd3d400b800
>>>>> nid=0x46c0 waiting on condition [0x00007fd3b8bb2000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e67194f0> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1576607881-147" #147 daemon prio=5 os_prio=0 tid=0x00007fd3ec5e5000
>>>>> nid=0x465a waiting on condition [0x00007fd3b8eb5000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e67194f0> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1576607881-145" #145 daemon prio=5 os_prio=0 tid=0x00007fd3e8092800
>>>>> nid=0x4626 waiting on condition [0x00007fd3b8cb3000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e67194f0> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1576607881-122" #122 daemon prio=5 os_prio=0 tid=0x00007fd3f401f000
>>>>> nid=0x41d0 waiting on condition [0x00007fd38feef000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e67194f0> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "ForkJoinPool-4-worker-7" #120 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd3b0048000 nid=0x41b7 waiting on condition
>>>>> [0x00007fd38fff0000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000fd545eb8> (a
>>>>> scala.concurrent.forkjoin.ForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkDriver-akka.actor.default-dispatcher-15" #119 daemon prio=5
>>>>> os_prio=0
>>>>> tid=0x00007fd3a4011000 nid=0x41b6 waiting on condition
>>>>> [0x00007fd3b8400000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e64f49f8> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "qtp1576607881-115" #115 daemon prio=5 os_prio=0 tid=0x00007fd3dc018000
>>>>> nid=0x417f waiting on condition [0x00007fd398935000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e67194f0> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "task-result-getter-3" #107 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd3b003f800
>>>>> nid=0x409c waiting on condition [0x00007fd3902f1000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e66e9a00> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "task-result-getter-2" #106 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd3b003d000
>>>>> nid=0x409b waiting on condition [0x00007fd3903f2000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e66e9a00> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "task-result-getter-1" #105 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd3b003b800
>>>>> nid=0x409a waiting on condition [0x00007fd3904f3000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e66e9a00> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "task-result-getter-0" #104 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd3b0030800
>>>>> nid=0x4098 waiting on condition [0x00007fd3905f4000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e66e9a00> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "threadDeathWatcher-2-1" #103 daemon prio=1 os_prio=0
>>>>> tid=0x00007fd3cc00c800
>>>>> nid=0x4093 waiting on condition [0x00007fd3936f8000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-server-3" #81 daemon prio=5 os_prio=0 tid=0x00007fd3a000a800
>>>>> nid=0x4092 runnable [0x00007fd398b37000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e685f320> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x00000000e685f380> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e685f288> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-server-2" #80 daemon prio=5 os_prio=0 tid=0x00007fd3a0005800
>>>>> nid=0x4091 runnable [0x00007fd398a36000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e6871c30> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x00000000e6872ca0> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e6871b98> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-server-1" #79 daemon prio=5 os_prio=0 tid=0x00007fd3a0007800
>>>>> nid=0x4090 runnable [0x00007fd39903a000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e6873520> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x00000000e6873580> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e6873488> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "org.apache.hadoop.hdfs.PeerCache@3cc2595a" #101 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd3e0011800 nid=0x4056 waiting on condition
>>>>> [0x00007fd398733000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
>>>>> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
>>>>> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "ForkJoinPool-3-worker-1" #100 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd3dc001800 nid=0x4055 waiting on condition
>>>>> [0x00007fd398834000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e7f19070> (a
>>>>> scala.concurrent.forkjoin.ForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkDriver-akka.actor.default-dispatcher-14" #96 daemon prio=5
>>>>> os_prio=0
>>>>> tid=0x00007fd3a400d800 nid=0x4051 waiting on condition
>>>>> [0x00007fd398e38000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e64f49f8> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "IPC Parameter Sending Thread #1" #95 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd432c05000 nid=0x404f waiting on condition
>>>>> [0x00007fd398f39000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e66fe220> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>>
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "BoneCP-pool-watch-thread" #93 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd43297c800 nid=0x4042 waiting on condition
>>>>> [0x00007fd39973b000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e740fa58> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>>>>> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "BoneCP-keep-alive-scheduler" #92 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd4329ac000 nid=0x4041 waiting on condition
>>>>> [0x00007fd39983c000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e740fce8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "com.google.common.base.internal.Finalizer" #91 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd4329aa000 nid=0x4040 in Object.wait() [0x00007fd39993d000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>>> - locked <0x00000000e740ff48> (a java.lang.ref.ReferenceQueue$Lock)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>>>> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "BoneCP-pool-watch-thread" #90 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd4328e6800 nid=0x403b waiting on condition
>>>>> [0x00007fd399c3e000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e737a0c8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>>>>> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "BoneCP-keep-alive-scheduler" #89 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd4328eb800 nid=0x403a waiting on condition
>>>>> [0x00007fd399d3f000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e737add8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "com.google.common.base.internal.Finalizer" #88 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd432532000 nid=0x4039 in Object.wait() [0x00007fd399e40000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>>> - locked <0x00000000e737b848> (a java.lang.ref.ReferenceQueue$Lock)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>>>> at com.google.common.base.internal.Finalizer.run(Finalizer.java:131)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Abandoned connection cleanup thread" #87 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd431b97000 nid=0x4036 in Object.wait() [0x00007fd39bdfe000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>>> - locked <0x00000000e7125130> (a java.lang.ref.ReferenceQueue$Lock)
>>>>> at
>>>>>
>>>>> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
>>>>>
>>>>> "ResponseProcessor for block
>>>>> BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #86 daemon
>>>>> prio=5 os_prio=0 tid=0x00007fd3a804d800 nid=0x3fe0 runnable
>>>>> [0x00007fd39a941000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e67d6048> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000000e67d6038> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e67d5e10> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
>>>>> at
>>>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>>>>> at
>>>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>>>>> at
>>>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
>>>>> at java.io.FilterInputStream.read(FilterInputStream.java:83)
>>>>> at java.io.FilterInputStream.read(FilterInputStream.java:83)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1985)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:808)
>>>>>
>>>>> "SparkListenerBus" #23 daemon prio=5 os_prio=0 tid=0x00007fd43258d000
>>>>> nid=0x3fda waiting on condition [0x00007fd39aa42000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e58145f0> (a
>>>>> java.util.concurrent.Semaphore$NonfairSync)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:62)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>>>>> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>>>>> at
>>>>>
>>>>> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
>>>>>
>>>>> "Spark Context Cleaner" #85 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd43258c000
>>>>> nid=0x3fd9 in Object.wait() [0x00007fd39ab43000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>>> - locked <0x00000000e6995428> (a java.lang.ref.ReferenceQueue$Lock)
>>>>> at
>>>>>
>>>>> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
>>>>> at
>>>>>
>>>>> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>>>>> at
>>>>>
>>>>> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
>>>>> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>>>>> at
>>>>>
>>>>> org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
>>>>> at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
>>>>>
>>>>> "DataStreamer for file
>>>>> /spark-logs/application_1427416187821_0004.inprogress
>>>>> block BP-1751822308-10.209.23.74-1427415953269:blk_1073747523_6699" #84
>>>>> daemon prio=5 os_prio=0 tid=0x00007fd43210a000 nid=0x3fba in
>>>>> Object.wait()
>>>>> [0x00007fd39ae44000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:507)
>>>>> - locked <0x00000000e6994f30> (a java.util.LinkedList)
>>>>>
>>>>> "metrics-ganglia-reporter-1-thread-1" #83 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd430f54800 nid=0x3fb8 waiting on condition
>>>>> [0x00007fd39af45000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e6995e88> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "shuffle-server-0" #78 daemon prio=5 os_prio=0 tid=0x00007fd4319f2800
>>>>> nid=0x3fb7 runnable [0x00007fd39b246000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e6886b58> (a
>>>>> io.netty.channel.nio.SelectedSelectionKeySet)
>>>>> - locked <0x00000000e689e610> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e6886ac0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>>>>> at
>>>>>
>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Yarn application state monitor" #73 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd4314ce800 nid=0x3fb4 waiting on condition
>>>>> [0x00007fd39b8fb000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>>
>>>>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:146)
>>>>>
>>>>> "Attach Listener" #72 daemon prio=9 os_prio=0 tid=0x00007fd3f4001000
>>>>> nid=0x3f77 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Thread-42" #70 daemon prio=5 os_prio=0 tid=0x00007fd41c131800
>>>>> nid=0x3e6f
>>>>> waiting on condition [0x00007fd39b547000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e677ae98> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "LeaseRenewer:hadoop@10.209.23.74:9000" #68 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd432896800 nid=0x3e6a waiting on condition
>>>>> [0x00007fd39b9fc000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
>>>>> at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
>>>>> at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Thread-36" #64 daemon prio=5 os_prio=0 tid=0x00007fd432850000
>>>>> nid=0x3e5e
>>>>> runnable [0x00007fd3b80ff000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "IPC Client (1489870911) connection to /10.209.23.74:9022 from hadoop"
>>>>> #62
>>>>> daemon prio=5 os_prio=0 tid=0x00007fd4326d3800 nid=0x3e43 in
>>>>> Object.wait()
>>>>> [0x00007fd3b8501000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
>>>>> - locked <0x00000000e66af0f8> (a
>>>>> org.apache.hadoop.ipc.Client$Connection)
>>>>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
>>>>>
>>>>> "dag-scheduler-event-loop" #61 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd4325f5800 nid=0x3e39 waiting on condition
>>>>> [0x00007fd3b8602000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e66e9230> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
>>>>> at
>>>>>
>>>>> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
>>>>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
>>>>>
>>>>> "Timer-0" #60 daemon prio=5 os_prio=0 tid=0x00007fd43253f000 nid=0x3e2f
>>>>> in
>>>>> Object.wait() [0x00007fd3b8703000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x00000000e66e8190> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "qtp1576607881-56" #56 daemon prio=5 os_prio=0 tid=0x00007fd432481000
>>>>> nid=0x3e2b waiting on condition [0x00007fd3b8db4000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e67194f0> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1576607881-54 Acceptor0 SelectChannelConnector@0.0.0.0:4040" #54
>>>>> daemon
>>>>> prio=5 os_prio=0 tid=0x00007fd43247e800 nid=0x3e29 runnable
>>>>> [0x00007fd3b8fb6000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>>> at
>>>>>
>>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
>>>>> at
>>>>>
>>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>>>>> - locked <0x00000000e671a620> (a java.lang.Object)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1576607881-52 Selector0" #52 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd432415000 nid=0x3e27 runnable [0x00007fd3b91b8000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e6719030> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000000e6719020> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e6718ef8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "SPARK_CONTEXT cleanup timer" #50 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd432237000 nid=0x3e20 in Object.wait() [0x00007fd3b96d6000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x00000000e66609f0> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "qtp906838480-49" #49 daemon prio=5 os_prio=0 tid=0x00007fd431ad4000
>>>>> nid=0x3e15 waiting on condition [0x00007fd3b99d7000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp906838480-48" #48 daemon prio=5 os_prio=0 tid=0x00007fd431ad2800
>>>>> nid=0x3e14 waiting on condition [0x00007fd3b9ad8000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp906838480-47" #47 daemon prio=5 os_prio=0 tid=0x00007fd431ad0800
>>>>> nid=0x3e13 waiting on condition [0x00007fd3b9bd9000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp906838480-46" #46 daemon prio=5 os_prio=0 tid=0x00007fd431ace800
>>>>> nid=0x3e12 waiting on condition [0x00007fd3b9cda000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp906838480-45" #45 daemon prio=5 os_prio=0 tid=0x00007fd431acc800
>>>>> nid=0x3e11 waiting on condition [0x00007fd3b9ddb000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp906838480-44" #44 daemon prio=5 os_prio=0 tid=0x00007fd431aca800
>>>>> nid=0x3e10 waiting on condition [0x00007fd3b9edc000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp906838480-43" #43 daemon prio=5 os_prio=0 tid=0x00007fd431ac9000
>>>>> nid=0x3e0f waiting on condition [0x00007fd3b9fdd000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e6539dc8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp906838480-42 Acceptor0 SocketConnector@0.0.0.0:53920" #42 daemon
>>>>> prio=5
>>>>> os_prio=0 tid=0x00007fd431abf000 nid=0x3e0e runnable
>>>>> [0x00007fd3ba0de000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at
>>>>> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:513)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "BROADCAST_VARS cleanup timer" #40 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd431a45000 nid=0x3e0d in Object.wait() [0x00007fd3ba1df000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x00000000e6601270> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "BLOCK_MANAGER cleanup timer" #39 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd431a22800 nid=0x3e0c in Object.wait() [0x00007fd3ba2e0000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x00000000e6600ca8> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "MAP_OUTPUT_TRACKER cleanup timer" #37 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd4317ea800 nid=0x3e0b in Object.wait() [0x00007fd3ba3e1000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:526)
>>>>> - locked <0x00000000e65380d8> (a java.util.TaskQueue)
>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>
>>>>> "New I/O server boss #6" #36 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd39c04b800
>>>>> nid=0x3df8 runnable [0x00007fd3ba6e2000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e6543220> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000000e6543210> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e6542e58> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #5" #35 daemon prio=5 os_prio=0 tid=0x00007fd39c049800
>>>>> nid=0x3df7 runnable [0x00007fd3ba7e3000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e6541ee8> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000000e6541ed8> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e6541dc0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #4" #34 daemon prio=5 os_prio=0 tid=0x00007fd39c04d800
>>>>> nid=0x3df6 runnable [0x00007fd3ba8e4000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e6542578> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000000e6542568> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e6542450> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O boss #3" #33 daemon prio=5 os_prio=0 tid=0x00007fd39c08c000
>>>>> nid=0x3df5 runnable [0x00007fd3ba9e5000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e657cbc8> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000000e657cbb8> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e657ca90> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #2" #31 daemon prio=5 os_prio=0 tid=0x00007fd39c029800
>>>>> nid=0x3df4 runnable [0x00007fd3baae6000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e64badd0> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000000e64badc0> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e64bac98> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "New I/O worker #1" #30 daemon prio=5 os_prio=0 tid=0x00007fd39c047000
>>>>> nid=0x3df3 runnable [0x00007fd3babe7000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>>>>> - locked <0x00000000e64bb368> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000000e64bb358> (a java.util.Collections$UnmodifiableSet)
>>>>> - locked <0x00000000e64bb240> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "sparkDriver-akka.remote.default-remote-dispatcher-6" #29 daemon prio=5
>>>>> os_prio=0 tid=0x00007fd4319dd800 nid=0x3dec waiting on condition
>>>>> [0x00007fd3baef9000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e64bbb40> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkDriver-akka.remote.default-remote-dispatcher-5" #28 daemon prio=5
>>>>> os_prio=0 tid=0x00007fd3ac016000 nid=0x3de8 waiting on condition
>>>>> [0x00007fd3baffa000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e64bbb40> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkDriver-akka.actor.default-dispatcher-4" #27 daemon prio=5
>>>>> os_prio=0
>>>>> tid=0x00007fd431f99000 nid=0x3de0 waiting on condition
>>>>> [0x00007fd3bb0fb000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e64f49f8> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkDriver-akka.actor.default-dispatcher-3" #26 daemon prio=5
>>>>> os_prio=0
>>>>> tid=0x00007fd431f96800 nid=0x3ddf waiting on condition
>>>>> [0x00007fd3bb1fc000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e64f49f8> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkDriver-akka.actor.default-dispatcher-2" #25 daemon prio=5
>>>>> os_prio=0
>>>>> tid=0x00007fd431f86000 nid=0x3dde waiting on condition
>>>>> [0x00007fd3bb2fd000]
>>>>>   java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e64f49f8> (a
>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>>>>> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>>>>> at
>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>> at
>>>>>
>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>>>
>>>>> "sparkDriver-scheduler-1" #24 daemon prio=5 os_prio=0
>>>>> tid=0x00007fd431f0e800
>>>>> nid=0x3ddd waiting on condition [0x00007fd3bb5fe000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>>>>> at
>>>>>
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
>>>>> at
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1423016050-21" #21 daemon prio=5 os_prio=0 tid=0x00007fd430fc1800
>>>>> nid=0x3d76 waiting on condition [0x00007fd3f9ad7000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1423016050-20" #20 daemon prio=5 os_prio=0 tid=0x00007fd430fc0000
>>>>> nid=0x3d75 waiting on condition [0x00007fd40c15d000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1423016050-19" #19 daemon prio=5 os_prio=0 tid=0x00007fd430fbc800
>>>>> nid=0x3d74 waiting on condition [0x00007fd40c25e000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1423016050-18" #18 daemon prio=5 os_prio=0 tid=0x00007fd430fba800
>>>>> nid=0x3d73 waiting on condition [0x00007fd40c35f000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1423016050-17" #17 daemon prio=5 os_prio=0 tid=0x00007fd430fb9000
>>>>> nid=0x3d72 waiting on condition [0x00007fd40c460000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1423016050-16" #16 daemon prio=5 os_prio=0 tid=0x00007fd430fb7000
>>>>> nid=0x3d71 waiting on condition [0x00007fd40c561000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1423016050-15" #15 daemon prio=5 os_prio=0 tid=0x00007fd430fb6000
>>>>> nid=0x3d70 waiting on condition [0x00007fd40c662000]
>>>>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000000e42808f8> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>>
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "qtp1423016050-14 Acceptor0 SocketConnector@0.0.0.0:53131" #14 daemon
>>>>> prio=5
>>>>> os_prio=0 tid=0x00007fd430fb5000 nid=0x3d6f runnable
>>>>> [0x00007fd40c966000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at
>>>>> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:513)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>>>> at
>>>>>
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd4300d1000
>>>>> nid=0x3d67 runnable [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd4300c3800
>>>>> nid=0x3d66 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd4300c2000
>>>>> nid=0x3d65 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd4300bf000
>>>>> nid=0x3d64 waiting on condition [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd4300bd800
>>>>> nid=0x3d63 runnable [0x0000000000000000]
>>>>>   java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd430085800 nid=0x3d62
>>>>> in
>>>>> Object.wait() [0x00007fd40f28e000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>>>>> - locked <0x00000000e000d1c8> (a java.lang.ref.ReferenceQueue$Lock)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>>>
>>>>> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd430083800
>>>>> nid=0x3d61 in Object.wait() [0x00007fd40f38f000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>>> - locked <0x00000000e000d438> (a java.lang.ref.Reference$Lock)
>>>>>
>>>>> "main" #1 prio=5 os_prio=0 tid=0x00007fd430011000 nid=0x3d5b in
>>>>> Object.wait() [0x00007fd438220000]
>>>>>   java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:502)
>>>>> at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
>>>>> - locked <0x00000000fda1a8c0> (a org.apache.spark.scheduler.JobWaiter)
>>>>> at
>>>>> org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1465)
>>>>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1534)
>>>>> at org.apache.spark.rdd.RDD.reduce(RDD.scala:900)
>>>>> at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1240)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.TakeOrdered.collectData(basicOperators.scala:137)
>>>>> at
>>>>>
>>>>> org.apache.spark.sql.execution.TakeOrdered.executeCollect(basicOperators.scala:141)
>>>>> at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:815)
>>>>> at org.apache.spark.sql.DataFrame.head(DataFrame.scala:758)
>>>>> at org.apache.spark.sql.DataFrame.take(DataFrame.scala:809)
>>>>> at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:178)
>>>>> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:314)
>>>>> at org.apache.spark.sql.DataFrame.show(DataFrame.scala:320)
>>>>> at
>>>>>
>>>>> $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
>>>>> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
>>>>> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
>>>>> at $line24.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
>>>>> at $line24.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
>>>>> at $line24.$read$$iwC$$iwC$$iwC.<init>(<console>:33)
>>>>> at $line24.$read$$iwC$$iwC.<init>(<console>:35)
>>>>> at $line24.$read$$iwC.<init>(<console>:37)
>>>>> at $line24.$read.<init>(<console>:39)
>>>>> at $line24.$read$.<init>(<console>:43)
>>>>> at $line24.$read$.<clinit>(<console>)
>>>>> at $line24.$eval$.<init>(<console>:7)
>>>>> at $line24.$eval$.<clinit>(<console>)
>>>>> at $line24.$eval.$print(<console>)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>> at
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>>> at
>>>>>
>>>>> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
>>>>> at
>>>>>
>>>>> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
>>>>> at
>>>>> org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
>>>>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
>>>>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
>>>>> at
>>>>> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
>>>>> at
>>>>>
>>>>> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
>>>>> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
>>>>> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
>>>>> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
>>>>> at
>>>>>
>>>>> org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
>>>>> at
>>>>>
>>>>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
>>>>> at
>>>>>
>>>>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>>>>> at
>>>>>
>>>>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
>>>>> at
>>>>>
>>>>> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
>>>>> at
>>>>>
>>>>> org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
>>>>> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
>>>>> at org.apache.spark.repl.Main$.main(Main.scala:31)
>>>>> at org.apache.spark.repl.Main.main(Main.scala)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>> at
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>>> at
>>>>>
>>>>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
>>>>> at
>>>>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
>>>>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
>>>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
>>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>>
>>>>> "VM Thread" os_prio=0 tid=0x00007fd43007e800 nid=0x3d60 runnable
>>>>>
>>>>> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd430026000
>>>>> nid=0x3d5c
>>>>> runnable
>>>>>
>>>>> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd430027800
>>>>> nid=0x3d5d
>>>>> runnable
>>>>>
>>>>> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd430029800
>>>>> nid=0x3d5e
>>>>> runnable
>>>>>
>>>>> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd43002b000
>>>>> nid=0x3d5f
>>>>> runnable
>>>>>
>>>>> "VM Periodic Task Thread" os_prio=0 tid=0x00007fd4300d3800 nid=0x3d68
>>>>> waiting on condition
>>>>>
>>>>> JNI global references: 420
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Mar 26, 2015 at 5:06 PM, Michael Armbrust
>>>>> <michael@databricks.com>
>>>>> wrote:
>>>>>>
>>>>>> Is it possible to jstack the executors and see where they are hanging?
>>>>>>
>>>>>> On Thu, Mar 26, 2015 at 2:02 PM, Jon Chase <jon.chase@gmail.com> wrote:
>>>>>>>
>>>>>>> Spark 1.3.0 on YARN (Amazon EMR), cluster of 10 m3.2xlarge (8cpu,
>>>>>>> 30GB),
>>>>>>> executor memory 20GB, driver memory 10GB
>>>>>>>
>>>>>>> I'm using Spark SQL, mainly via spark-shell, to query 15GB of data
>>>>>>> spread
>>>>>>> out over roughly 2,000 Parquet files and my queries frequently hang.
>>>>>>> Simple
>>>>>>> queries like "select count(*) from ..." on the entire data set work
>>>>>>> ok.
>>>>>>> Slightly more demanding ones with group by's and some aggregate
>>>>>>> functions
>>>>>>> (percentile_approx, avg, etc.) work ok as well, as long as I have some
>>>>>>> criteria in my where clause to keep the number of rows down.
>>>>>>>
>>>>>>> Once I hit some limit on query complexity and rows processed, my
>>>>>>> queries
>>>>>>> start to hang.  I've left them for up to an hour without seeing any
>>>>>>> progress.  No OOM's either - the job is just stuck.
>>>>>>>
>>>>>>> I've tried setting spark.sql.shuffle.partitions to 400 and even 800,
>>>>>>> but
>>>>>>> with the same results: usually near the end of the tasks (like 780 of
>>>>>>> 800
>>>>>>> complete), progress just stops:
>>>>>>>
>>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 788.0
>>>>>>> in
>>>>>>> stage 1.0 (TID 1618) in 800 ms on
>>>>>>> ip-10-209-22-211.eu-west-1.compute.internal (748/800)
>>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 793.0
>>>>>>> in
>>>>>>> stage 1.0 (TID 1623) in 622 ms on
>>>>>>> ip-10-105-12-41.eu-west-1.compute.internal
>>>>>>> (749/800)
>>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 797.0
>>>>>>> in
>>>>>>> stage 1.0 (TID 1627) in 616 ms on
>>>>>>> ip-10-90-2-201.eu-west-1.compute.internal
>>>>>>> (750/800)
>>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 799.0
>>>>>>> in
>>>>>>> stage 1.0 (TID 1629) in 611 ms on
>>>>>>> ip-10-90-2-201.eu-west-1.compute.internal
>>>>>>> (751/800)
>>>>>>> 15/03/26 20:53:29 INFO scheduler.TaskSetManager: Finished task 795.0
>>>>>>> in
>>>>>>> stage 1.0 (TID 1625) in 669 ms on
>>>>>>> ip-10-105-12-41.eu-west-1.compute.internal
>>>>>>> (752/800)
>>>>>>>
>>>>>>> ^^^^^^^ this is where it stays forever
>>>>>>>
>>>>>>> Looking at the Spark UI, several of the executors still list active
>>>>>>> tasks.  I do see that the Shuffle Read for executors that don't have
>>>>>>> any
>>>>>>> tasks remaining is around 100MB, whereas it's more like 10MB for the
>>>>>>> executors that still have tasks.
>>>>>>>
>>>>>>> The first stage, mapPartitions, always completes fine.  It's the
>>>>>>> second
>>>>>>> stage (takeOrdered), that hangs.
>>>>>>>
>>>>>>> I've had this issue in 1.2.0 and 1.2.1 as well as 1.3.0.  I've also
>>>>>>> encountered it when using JSON files (instead of Parquet).
>>>>>>>
>>>>>>> Thoughts?  I'm blocked on using Spark SQL b/c most of the queries I do
>>>>>>> are having this issue.
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Christian Perez
>>>> Silicon Valley Data Science
>>>> Data Analyst
>>>> christian@svds.com
>>>> @cp_phd
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>
>>
>>
>>
>> --
>> Christian Perez
>> Silicon Valley Data Science
>> Data Analyst
>> christian@svds.com
>> @cp_phd
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>



-- 
Christian Perez
Silicon Valley Data Science
Data Analyst
christian@svds.com
@cp_phd

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message