spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Sharma <resolve...@gmail.com>
Subject Spark Null Pointer Exception
Date Wed, 30 Jun 2021 20:47:40 GMT
Hi , I am using spark 2.7 version with scala. I am calling a method as
below

1. val rddBacklog = spark.sparkContext.parallelize(MAs) // MA is list
of say city

2. rddBacklog.foreach(ma => doAlloc3Daily(ma, fteReview.forecastId,
startYear, endYear))

3.doAlloc3Daily method just doing a database call and doing some scala
calculation (no rdd or dataframe)


Line number 2 I  am getting below  nullpointer intermittently on
cluster but never on local.

java.lang.NullPointerException
	at sparkStreaming.CalculateFteReview.doAlloc3Daily(CalculateFteReview.scala:1307)
	at sparkStreaming.CalculateFteReview$$anonfun$getNewDistribution$2.apply(CalculateFteReview.scala:1199)
	at sparkStreaming.CalculateFteReview$$anonfun$getNewDistribution$2.apply(CalculateFteReview.scala:1199)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
	at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:927)
	at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:927)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:109)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


Thanks

Amit

Mime
View raw message