spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <hol...@pigscanfly.ca>
Subject Re: Spark Streaming testing strategies
Date Tue, 10 Mar 2015 20:24:23 GMT
On Tue, Mar 10, 2015 at 1:18 PM, Marcin Kuthan <marcin.kuthan@gmail.com>
wrote:

> Hi Holden
>
> Thanks Holden for pointing me the package. Indeed StreamingSuiteBase
> trait hides a lot, especially regarding clock manipulation. Did you
> encounter problems with concurrent tests execution from SBT
> (SPARK-2243)? I had to disable parallel execution and configure SBT to
> use separate JVM for tests execution (fork).
>
Yah, I haven't used parallel execution with this testing trait, I can look
into it some more.

>
> BTW. I added samples for SparkSQL as well.
>
Oh awesome :)

>
> I would expect base trait for testing purposes in spark distribution.
> ManualClock should be exposed as well. And some documentation how to
> configure SBT to avoid problems with multiple spark contexts. I'm
> going to create improvement proposal on Spark issue tracker about it.
>
Right now I think a package is probably a good place for this to live since
the internal Spark testing code is changing/evolving rapidly, but I think
once we have the trait fleshed out a bit more we could see if there is
enough interest to try and merge it in (just my personal thoughts).


>
>
>
> On 1 March 2015 at 18:49, Holden Karau <holden@pigscanfly.ca> wrote:
> >
> > There is also the Spark Testing Base package which is on
> spark-packages.org and hides the ugly bits (it's based on the existing
> streaming test code but I cleaned it up a bit to try and limit the number
> of internals it was touching).
> >
> >
> > On Sunday, March 1, 2015, Marcin Kuthan <marcin.kuthan@gmail.com> wrote:
> >>
> >> I have started using Spark and Spark Streaming and I'm wondering how do
> you test your applications? Especially Spark Streaming application with
> window based transformations.
> >>
> >> After some digging I found ManualClock class to take full control over
> stream processing. Unfortunately the class is not available outside
> spark.streaming package. Are you going to expose the class for other
> developers as well? Now I have to use my custom wrapper under
> spark.streaming package.
> >>
> >> My Spark and Spark Streaming unit tests strategies are documented here:
> >> http://mkuthan.github.io/blog/2015/03/01/spark-unit-testing/
> >>
> >> Your feedback is more than appreciated.
> >>
> >> Marcin
> >>
> >
> >
> > --
> > Cell : 425-233-8271
>



-- 
Cell : 425-233-8271

Mime
View raw message