spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcin Kuthan <>
Subject Re: Spark Streaming testing strategies
Date Tue, 10 Mar 2015 20:18:19 GMT
Hi Holden

Thanks Holden for pointing me the package. Indeed StreamingSuiteBase
trait hides a lot, especially regarding clock manipulation. Did you
encounter problems with concurrent tests execution from SBT
(SPARK-2243)? I had to disable parallel execution and configure SBT to
use separate JVM for tests execution (fork).

BTW. I added samples for SparkSQL as well.

I would expect base trait for testing purposes in spark distribution.
ManualClock should be exposed as well. And some documentation how to
configure SBT to avoid problems with multiple spark contexts. I'm
going to create improvement proposal on Spark issue tracker about it.

On 1 March 2015 at 18:49, Holden Karau <> wrote:
> There is also the Spark Testing Base package which is on and hides
the ugly bits (it's based on the existing streaming test code but I cleaned it up a bit to
try and limit the number of internals it was touching).
> On Sunday, March 1, 2015, Marcin Kuthan <> wrote:
>> I have started using Spark and Spark Streaming and I'm wondering how do you test
your applications? Especially Spark Streaming application with window based transformations.
>> After some digging I found ManualClock class to take full control over stream processing.
Unfortunately the class is not available outside spark.streaming package. Are you going to
expose the class for other developers as well? Now I have to use my custom wrapper under spark.streaming
>> My Spark and Spark Streaming unit tests strategies are documented here:
>> Your feedback is more than appreciated.
>> Marcin
> --
> Cell : 425-233-8271

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message