spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <hol...@pigscanfly.ca>
Subject Re: Spark Streaming testing strategies
Date Sun, 01 Mar 2015 17:49:34 GMT
There is also the Spark Testing Base package which is on spark-packages.org
and hides the ugly bits (it's based on the existing streaming test code but
I cleaned it up a bit to try and limit the number of internals it was
touching).

On Sunday, March 1, 2015, Marcin Kuthan <marcin.kuthan@gmail.com> wrote:

> I have started using Spark and Spark Streaming and I'm wondering how do
> you test your applications? Especially Spark Streaming application with
> window based transformations.
>
> After some digging I found ManualClock class to take full control over
> stream processing. Unfortunately the class is not available outside
> spark.streaming package. Are you going to expose the class for other
> developers as well? Now I have to use my custom wrapper under
> spark.streaming package.
>
> My Spark and Spark Streaming unit tests strategies are documented here:
> http://mkuthan.github.io/blog/2015/03/01/spark-unit-testing/
>
> Your feedback is more than appreciated.
>
> Marcin
>
>

-- 
Cell : 425-233-8271

Mime
View raw message