spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcin Kuthan <marcin.kut...@gmail.com>
Subject Spark Streaming testing strategies
Date Sun, 01 Mar 2015 09:13:49 GMT
I have started using Spark and Spark Streaming and I'm wondering how do you
test your applications? Especially Spark Streaming application with window
based transformations.

After some digging I found ManualClock class to take full control over
stream processing. Unfortunately the class is not available outside
spark.streaming package. Are you going to expose the class for other
developers as well? Now I have to use my custom wrapper under
spark.streaming package.

My Spark and Spark Streaming unit tests strategies are documented here:
http://mkuthan.github.io/blog/2015/03/01/spark-unit-testing/

Your feedback is more than appreciated.

Marcin

Mime
View raw message