There is also the Spark Testing Base package which is on and hides the ugly bits (it's based on the existing streaming test code but I cleaned it up a bit to try and limit the number of internals it was touching). 

On Sunday, March 1, 2015, Marcin Kuthan <> wrote:
I have started using Spark and Spark Streaming and I'm wondering how do you test your applications? Especially Spark Streaming application with window based transformations.

After some digging I found ManualClock class to take full control over stream processing. Unfortunately the class is not available outside spark.streaming package. Are you going to expose the class for other developers as well? Now I have to use my custom wrapper under spark.streaming package.

My Spark and Spark Streaming unit tests strategies are documented here:

Your feedback is more than appreciated.


Cell : 425-233-8271