spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <>
Subject Re: Adding Spark Testing functionality
Date Mon, 12 Oct 2015 18:37:06 GMT
So here is a quick description of the current testing bits (I can expand on
it if people are interested) .

On Tue, Oct 6, 2015 at 3:49 PM, Holden Karau <> wrote:

> I'll put together a google doc and send that out (in the meantime a quick
> guide of sort of how the current package can be used is in the blog post I
> did at
> )  If people think its better to keep as a package I am of course happy to
> keep doing that. It feels a little strange to have something as core as
> being able to test your code live outside.
> On Tue, Oct 6, 2015 at 3:44 PM, Patrick Wendell <>
> wrote:
>> Hey Holden,
>> It would be helpful if you could outline the set of features you'd
>> imagine being part of Spark in a short doc. I didn't see a README on the
>> existing repo, so it's hard to know exactly what is being proposed.
>> As a general point of process, we've typically avoided merging modules
>> into Spark that can exist outside of the project. A testing utility package
>> that is based on Spark's public API's seems like a really useful thing for
>> the community, but it does seem like a good fit for a package library. At
>> least, this is my first question after taking a look at the project.
>> In any case, getting some high level view of the functionality you
>> imagine would be helpful to give more detailed feedback.
>> - Patrick
>> On Tue, Oct 6, 2015 at 3:12 PM, Holden Karau <>
>> wrote:
>>> Hi Spark Devs,
>>> So this has been brought up a few times before, and generally on the
>>> user list people get directed to use spark-testing-base. I'd like to start
>>> moving some of spark-testing-base's functionality into Spark so that people
>>> don't need a library to do what is (hopefully :p) a very common requirement
>>> across all Spark projects.
>>> To that end I was wondering what peoples thoughts are on where this
>>> should live inside of Spark. I was thinking it could either be a separate
>>> testing project (like sql or similar), or just put the bits to enable
>>> testing inside of each relevant project.
>>> I was also thinking it probably makes sense to only move the unit
>>> testing parts at the start and leave things like integration testing in a
>>> testing project since that could vary depending on the users environment.
>>> What are peoples thoughts?
>>> Cheers,
>>> Holden :)
> --
> Cell : 425-233-8271
> Twitter:
> Linked In:

Cell : 425-233-8271
Linked In:

View raw message