spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Tse <Mark....@D2L.com>
Subject Unit Testing Spark Transformations/Actions
Date Tue, 16 Jun 2015 16:19:40 GMT
Hi there,

I am looking to use Mockito to mock out some functionality while unit testing a Spark application.

I currently have code that happily runs on a cluster, but fails when I try to run unit tests
against it, throwing a "SparkException":

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed
1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost): java.lang.ClassCastException:
cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1.f$14
of type org.apache.spark.api.java.function.VoidFunction in instance of org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1
    at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2089)

(Full error/stacktrace and description on SO: http://stackoverflow.com/q/30871109/2687324).

Has anyone experienced this error before while unit testing?

Thanks,
Mark

Mime
View raw message