spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saket Kumar <>
Subject Unit testing: Mocking out Spark classes
Date Thu, 16 Oct 2014 13:07:36 GMT
Hello all,

I am trying to unit test my classes involved my Spark job. I am trying to
mock out the Spark classes (like SparkContext and Broadcast) so that I can
unit test my classes in isolation. However I have realised that these are
classes instead of traits. My first question is why?

It is quite hard to mock out classes using ScalaTest+ScalaMock as the
classes which need to be mocked out need to be annotated with
org.scalamock.annotation.mock as per
I cannot do that in my case as I am trying to mock out the spark classes.

Am I missing something? Is there a better way to do this?

    val sparkContext = mock[SparkInteraction]
    val trainingDatasetLoader = mock[DatasetLoader]
    val broadcastTrainingDatasetLoader = mock[Broadcast[DatasetLoader]]
    def transformerFunction(source: Iterator[(HubClassificationData,
String)]): Iterator[String] = {
    val classificationResultsRDD = mock[RDD[String]]
    val classificationResults = Array("","","")
    val inputRDD = mock[RDD[(HubClassificationData, String)]]


    val sparkInvoker = new SparkJobInvoker(sparkContext,



View raw message