spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcin Kuthan <>
Subject Re: Spark Streaming testing strategies
Date Tue, 10 Mar 2015 21:20:01 GMT
>> I would expect base trait for testing purposes in spark distribution.
>> ManualClock should be exposed as well. And some documentation how to
>> configure SBT to avoid problems with multiple spark contexts. I'm
>> going to create improvement proposal on Spark issue tracker about it.
> Right now I think a package is probably a good place for this to live since
> the internal Spark testing code is changing/evolving rapidly, but I think
> once we have the trait fleshed out a bit more we could see if there is
> enough interest to try and merge it in (just my personal thoughts).

You are right, let's wait for community feedback.

I'm quite new in Spark but it is strange for me that good support for
isolated testing is not first class citizen. In the Spark Programming
Guide the chapter about unit testing is not more than three sentences

Spring Framework taught me how important is the ability to run tests
from your IDE, and how important is tests execution performance (there
are also problems with parallelism under single JVM).

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message