spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <>
Subject Re: Adding Spark Testing functionality
Date Tue, 06 Oct 2015 22:44:12 GMT
Hey Holden,

It would be helpful if you could outline the set of features you'd imagine
being part of Spark in a short doc. I didn't see a README on the existing
repo, so it's hard to know exactly what is being proposed.

As a general point of process, we've typically avoided merging modules into
Spark that can exist outside of the project. A testing utility package that
is based on Spark's public API's seems like a really useful thing for the
community, but it does seem like a good fit for a package library. At
least, this is my first question after taking a look at the project.

In any case, getting some high level view of the functionality you imagine
would be helpful to give more detailed feedback.

- Patrick

On Tue, Oct 6, 2015 at 3:12 PM, Holden Karau <> wrote:

> Hi Spark Devs,
> So this has been brought up a few times before, and generally on the user
> list people get directed to use spark-testing-base. I'd like to start
> moving some of spark-testing-base's functionality into Spark so that people
> don't need a library to do what is (hopefully :p) a very common requirement
> across all Spark projects.
> To that end I was wondering what peoples thoughts are on where this should
> live inside of Spark. I was thinking it could either be a separate testing
> project (like sql or similar), or just put the bits to enable testing
> inside of each relevant project.
> I was also thinking it probably makes sense to only move the unit testing
> parts at the start and leave things like integration testing in a testing
> project since that could vary depending on the users environment.
> What are peoples thoughts?
> Cheers,
> Holden :)

View raw message