spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <>
Subject Re: Spark Streaming testing strategies
Date Sun, 01 Mar 2015 17:49:34 GMT
There is also the Spark Testing Base package which is on
and hides the ugly bits (it's based on the existing streaming test code but
I cleaned it up a bit to try and limit the number of internals it was

On Sunday, March 1, 2015, Marcin Kuthan <> wrote:

> I have started using Spark and Spark Streaming and I'm wondering how do
> you test your applications? Especially Spark Streaming application with
> window based transformations.
> After some digging I found ManualClock class to take full control over
> stream processing. Unfortunately the class is not available outside
> spark.streaming package. Are you going to expose the class for other
> developers as well? Now I have to use my custom wrapper under
> spark.streaming package.
> My Spark and Spark Streaming unit tests strategies are documented here:
> Your feedback is more than appreciated.
> Marcin

Cell : 425-233-8271

View raw message