spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: What this exception means? ConnectionManager: key already cancelled ?
Date Mon, 27 Oct 2014 16:02:22 GMT
Setting the following while creating the sparkContext will sort it out.

                .set("spark.core.connection.ack.wait.timeout","600")

.set("spark.akka.frameSize","50")

On 27 Oct 2014 21:15, "shahab" <shahab.mokari@gmail.com> wrote:

> Hi,
>
> I have a stand alone Spark Cluster, where worker and master reside on the
> same machine. I submit a job to the cluster, the job is executed for a
> while and suddenly I get this exception  with no additional trace.
>
> ConnectionManager: key already cancelled ?
> sun.nio.ch.SelectionKeyImpl@2490dce9
> java.nio.channels.CancelledKeyException at
> org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
>  at
> org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
>
>
> Any idea where should I look for the cause?
>
> best,
> /shahab
>
> This following is the part of printout from "driver application" logs:
>
> 14/10/27 15:21:15 INFO BlockManagerInfo: Removed broadcast_1_piece0 on
> ip-10-89-32-179.eu-west-1.compute.internal:40479 in memory (size: 3.4 KB,
> free: 1565.6 MB)
> 14/10/27 15:21:15 INFO ContextCleaner: Cleaned broadcast 1
> 14/10/27 15:21:15 INFO ShuffleBlockManager: Could not find files for
> shuffle 1 for deleting
> 14/10/27 15:21:15 INFO ContextCleaner: Cleaned shuffle 1
> 14/10/27 15:21:15 INFO ShuffleBlockManager: Could not find files for
> shuffle 0 for deleting
> 14/10/27 15:21:15 INFO ContextCleaner: Cleaned shuffle 0
> 14/10/27 15:21:15 INFO BlockManagerInfo: Removed taskresult_9 on
> ip-10-zz.xx-yy:40479 in memory (size: 24.1 MB, free: 1589.8 MB)
> 14/10/27 15:21:16 INFO DAGScheduler: Stage 7 (collect at
> TimeBenchmarking_SimpleModel.scala:55) finished in 3.209 s
> 14/10/27 15:21:16 INFO TaskSetManager: Finished task 0.0 in stage 7.0 (TID
> 9) in 2640 ms onip-10-zz.xx-yy (1/1)
> 14/10/27 15:21:16 INFO SparkContext: Job finished: collect at
> TimeBenchmarking_SimpleModel.scala:55, took 102.661420511 s
> 14/10/27 15:21:16 INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks
> have all completed, from pool
> 14/10/27 15:21:16 INFO SparkUI: Stopped Spark web UI at
> http://ip-10-zz.xx-yy:4040
> 14/10/27 15:21:16 INFO DAGScheduler: Stopping DAGScheduler
> 14/10/27 15:21:16 INFO SparkDeploySchedulerBackend: Shutting down all
> executors
> 14/10/27 15:21:16 INFO SparkDeploySchedulerBackend: Asking each executor
> to shut down
> 14/10/27 15:21:16 INFO ConnectionManager: Removing ReceivingConnection to
> ConnectionManagerId(ip-10-zz.xx-yy, 40479)
> 14/10/27 15:21:16 INFO ConnectionManager: Removing SendingConnection to
> ConnectionManagerId(ip-10-zz.xx-yy,40479)
> 14/10/27 15:21:16 INFO ConnectionManager: Removing SendingConnection to
> ConnectionManagerId(ip-10-zz.xx-yy,40479)
> 14/10/27 15:21:16 INFO ConnectionManager: Key not valid ?
> sun.nio.ch.SelectionKeyImpl@2490dce9
> 14/10/27 15:21:16 INFO ConnectionManager: key already cancelled ?
> sun.nio.ch.SelectionKeyImpl@2490dce9
> java.nio.channels.CancelledKeyException
>         at
> org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
>         at
> org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
> 14/10/27 15:21:17 INFO MapOutputTrackerMasterActor: MapOutputTrackerActor
> stopped!
> 14/10/27 15:21:17 INFO ConnectionManager: Selector thread was interrupted!
>

Mime
View raw message