spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wangfei (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-1361) DAGScheduler throws NullPointerException occasionally
Date Mon, 31 Mar 2014 09:18:16 GMT

    [ https://issues.apache.org/jira/browse/SPARK-1361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13955046#comment-13955046
] 

wangfei commented on SPARK-1361:
--------------------------------

this is an order issue of construction and starting of the taskScheduler and the dagScheduler
in SparkContext.  now taskScheduler start before  dagScheduler, so executor backend may register
executor even before dagScheduler(or the eventProcessActor) has not initialized. in this situation
it will throw NPE

> DAGScheduler throws NullPointerException occasionally
> -----------------------------------------------------
>
>                 Key: SPARK-1361
>                 URL: https://issues.apache.org/jira/browse/SPARK-1361
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 0.9.0
>            Reporter: wangfei
>             Fix For: 1.0.0
>
>
> DAGScheduler throws this NullPointerException below occasionally when running spark apps.
> java.lang.NullPointerException
> at org.apache.spark.scheduler.DAGScheduler.executorAdded(DAGScheduler.scala:186)
> at org.apache.spark.scheduler.TaskSchedulerImpl.executorAdded(TaskSchedulerImpl.scala:409)
> at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$resourceOffers$1.apply(TaskSchedulerImpl.scala:210)
> at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$resourceOffers$1.apply(TaskSchedulerImpl.scala:206)
> at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:73)
> at org.apache.spark.scheduler.TaskSchedulerImpl.resourceOffers(TaskSchedulerImpl.scala:206)
> at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverActor.makeOffers(CoarseGrainedSchedulerBackend.scala:130)
> at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverActor$$anonfun$receive$1.applyOrEls



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message