spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Rosen (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-12945) ERROR LiveListenerBus: Listener JobProgressListener threw an exception
Date Mon, 27 Feb 2017 19:46:46 GMT

    [ https://issues.apache.org/jira/browse/SPARK-12945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886397#comment-15886397
] 

Josh Rosen edited comment on SPARK-12945 at 2/27/17 7:46 PM:
-------------------------------------------------------------

I'm still seeing this error intermittently on Spark 2.0.2 clusters. Here's the stacktrace
as of that version:

{code}
"Listener JobProgressListener threw an exception
java.lang.NullPointerException
	at org.apache.spark.ui.jobs.JobProgressListener$$anonfun$onTaskEnd$1.apply(JobProgressListener.scala:365)
	at org.apache.spark.ui.jobs.JobProgressListener$$anonfun$onTaskEnd$1.apply(JobProgressListener.scala:364)
	at scala.collection.immutable.List.foreach(List.scala:318)
	at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
	at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
	at org.apache.spark.ui.jobs.JobProgressListener.onTaskEnd(JobProgressListener.scala:364)
	at org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:45)
	at org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
	at org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
	at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:63)
	at org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:36)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiveListenerBus.scala:94)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:78)
	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1252)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:77)"
{code}

Possibly related is an NPE in TaskUIData:

{code}
"Listener JobProgressListener threw an exception
java.lang.NullPointerException
	at org.apache.spark.ui.jobs.UIData$TaskUIData$$anonfun$dropInternalAndSQLAccumulables$1.apply(UIData.scala:169)
	at org.apache.spark.ui.jobs.UIData$TaskUIData$$anonfun$dropInternalAndSQLAccumulables$1.apply(UIData.scala:169)
	at scala.collection.TraversableLike$$anonfun$filter$1.apply(TraversableLike.scala:264)
	at scala.collection.immutable.List.foreach(List.scala:318)
	at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
	at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
	at scala.collection.TraversableLike$class.filter(TraversableLike.scala:263)
	at scala.collection.AbstractTraversable.filter(Traversable.scala:105)
	at org.apache.spark.ui.jobs.UIData$TaskUIData$.dropInternalAndSQLAccumulables(UIData.scala:168)
	at org.apache.spark.ui.jobs.UIData$TaskUIData.updateTaskInfo(UIData.scala:120)
	at org.apache.spark.ui.jobs.JobProgressListener.onTaskEnd(JobProgressListener.scala:401)
	at org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:45)
	at org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
	at org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
	at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:63)
	at org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:36)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiveListenerBus.scala:94)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:78)
	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1252)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:77)"
{code}

My hunch is that {{accumulables}} is somehow null.


was (Author: joshrosen):
I'm still seeing this error intermittently on Spark 2.0.3 clusters. Here's the stacktrace
as of that version:

{code}
"Listener JobProgressListener threw an exception
java.lang.NullPointerException
	at org.apache.spark.ui.jobs.JobProgressListener$$anonfun$onTaskEnd$1.apply(JobProgressListener.scala:365)
	at org.apache.spark.ui.jobs.JobProgressListener$$anonfun$onTaskEnd$1.apply(JobProgressListener.scala:364)
	at scala.collection.immutable.List.foreach(List.scala:318)
	at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
	at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
	at org.apache.spark.ui.jobs.JobProgressListener.onTaskEnd(JobProgressListener.scala:364)
	at org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:45)
	at org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
	at org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
	at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:63)
	at org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:36)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiveListenerBus.scala:94)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:78)
	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1252)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:77)"
{code}

Possibly related is an NPE in TaskUIData:

{code}
"Listener JobProgressListener threw an exception
java.lang.NullPointerException
	at org.apache.spark.ui.jobs.UIData$TaskUIData$$anonfun$dropInternalAndSQLAccumulables$1.apply(UIData.scala:169)
	at org.apache.spark.ui.jobs.UIData$TaskUIData$$anonfun$dropInternalAndSQLAccumulables$1.apply(UIData.scala:169)
	at scala.collection.TraversableLike$$anonfun$filter$1.apply(TraversableLike.scala:264)
	at scala.collection.immutable.List.foreach(List.scala:318)
	at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
	at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
	at scala.collection.TraversableLike$class.filter(TraversableLike.scala:263)
	at scala.collection.AbstractTraversable.filter(Traversable.scala:105)
	at org.apache.spark.ui.jobs.UIData$TaskUIData$.dropInternalAndSQLAccumulables(UIData.scala:168)
	at org.apache.spark.ui.jobs.UIData$TaskUIData.updateTaskInfo(UIData.scala:120)
	at org.apache.spark.ui.jobs.JobProgressListener.onTaskEnd(JobProgressListener.scala:401)
	at org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:45)
	at org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
	at org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
	at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:63)
	at org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:36)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiveListenerBus.scala:94)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:78)
	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1252)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:77)"
{code}

My hunch is that {{accumulables}} is somehow null.

> ERROR LiveListenerBus: Listener JobProgressListener threw an exception
> ----------------------------------------------------------------------
>
>                 Key: SPARK-12945
>                 URL: https://issues.apache.org/jira/browse/SPARK-12945
>             Project: Spark
>          Issue Type: Bug
>          Components: Web UI
>    Affects Versions: 1.6.0, 2.0.2
>         Environment: Linux, yarn-client
>            Reporter: Tristan
>            Priority: Minor
>
> Seeing this a lot; not sure if it is a problem or spurious error (I recall this was an
ignorable issue in previous version). The UI seems to be working fine:
> ERROR LiveListenerBus: Listener JobProgressListener threw an exception
> java.lang.NullPointerException
>         at org.apache.spark.ui.jobs.JobProgressListener$$anonfun$onTaskEnd$1.apply(JobProgressListener.scala:361)
>         at org.apache.spark.ui.jobs.JobProgressListener$$anonfun$onTaskEnd$1.apply(JobProgressListener.scala:360)
>         at scala.collection.immutable.List.foreach(List.scala:318)
>         at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>         at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
>         at org.apache.spark.ui.jobs.JobProgressListener.onTaskEnd(JobProgressListener.scala:360)
>         at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:42)
>         at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
>         at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
>         at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:55)
>         at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37)
>         at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(AsynchronousListenerBus.scala:80)
>         at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:65)
>         at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:65)
>         at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
>         at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:64)
>         at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1180)
>         at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message