spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mario Pastorelli <mario.pastore...@teralytics.ch>
Subject Re: Using FunSuite to test Spark throws NullPointerException
Date Mon, 06 Oct 2014 21:22:48 GMT
The problem was mine: I was using FunSuite in the wrong way. The test 
method of FunSuite registers a "test" to be triggered when the tests are 
running. My localTest is instead creating and stopping the SparkContext 
during the test registration and as result my SparkContext is stopped 
when the tests are run. The way to fix this is to wrap everything inside 
the test method:

import org.scalatest.FunSuite

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext


class SparkFunSuite extends FunSuite {

   def localTest(name : String)(f : SparkContext => Unit) : Unit = {
     this.test(name) {
       val conf = new SparkConf()
         .setAppName(name)
         .setMaster("local")
         .set("spark.default.parallelism", "1")
       val sc = new SparkContext(conf)
       try {
         f(sc)
       } finally {
         sc.stop()
       }
     }
   }

}

class MyTestSuite extends SparkFunSuite {

   localTest("My Spark test") { sc =>
     assertResult(2)(sc.parallelize(Seq(1,2,3)).filter(_ <= 2).map(_ + 
1).count)
   }
}

Thanks anyway


On 06.10.2014 20:13, Matei Zaharia wrote:
> Weird, it seems like this is trying to use the SparkContext before 
> it's initialized, or something like that. Have you tried unrolling 
> this into a single method? I wonder if you just have multiple versions 
> of these libraries on your classpath or something.
>
> Matei
>
> On Oct 4, 2014, at 1:40 PM, Mario Pastorelli 
> <mario.pastorelli@teralytics.ch 
> <mailto:mario.pastorelli@teralytics.ch>> wrote:
>
>> I would like to use FunSuite 
>> <http://doc.scalatest.org/2.2.1/index.html#org.scalatest.FunSuite> to 
>> test my Spark jobs by extending FunSuite with a new function, called 
>> |localTest|, that runs a test with a default SparkContext:
>>
>> |class  SparkFunSuite  extends  FunSuite  {
>>
>>    def  localTest(name:  String)(f:  SparkContext  =>  Unit)  :  Unit  =  {
>>      val  conf=  new  SparkConf().setAppName(name).setMaster("local")
>>      val  sc=  new  SparkContext(conf)
>>      try  {
>>        this.test(name)(f(sc))
>>      }  finally  {
>>        sc.stop
>>      }
>>    }
>> }|
>>
>> Then I can add tests easily to my testing suites:
>>
>> |class  MyTestSuite  extends  SparkFunSuite  {
>>
>>    localTest("My Spark test")  {  sc=>
>>      assertResult(2)(sc.parallelize(Seq(1,2,3)).filter(_<=  2).map(_+  1).count)
>>    }
>> }|
>>
>> The problem is that when I run the tests I get a |NullPointerException|:
>>
>> |[info]  MyTestSuite:
>> [info]  -  My  Spark  test***  FAILED***
>> [info]    java.lang.NullPointerException:
>> [info]    at org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1215)
>> [info]    at org.apache.spark.SparkContext.parallelize$default$2(SparkContext.scala:435)
>> [info]    atMyTestSuite$$anonfun$1.apply(FunSuiteTest.scala:24)
>> [info]    atMyTestSuite$$anonfun$1.apply(FunSuiteTest.scala:23)
>> [info]    atSparkFunSuite$$anonfun$localTest$1.apply$mcV$sp(FunSuiteTest.scala:13)
>> [info]    atSparkFunSuite$$anonfun$localTest$1.apply(FunSuiteTest.scala:13)
>> [info]    atSparkFunSuite$$anonfun$localTest$1.apply(FunSuiteTest.scala:13)
>> [info]    at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>> [info]    at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>> [info]    at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>> [info]    ...|
>>
>> What is causing the |NullPointerException|? More importantly, why I 
>> get this error while the testing suite of Spark, that does something 
>> similar, doesn't? Can somebody explain me the difference between them?
>>
>> I'm using Scala 2.10.4 with |spark-core| 1.0.2 and |scalatest| 2.2.2.
>>
>


Mime
View raw message