spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Adams <radams...@gmail.com>
Subject Re: testing frameworks
Date Tue, 12 Jun 2018 17:44:16 GMT
We use spark testing base for unit testing.  These tests execute on a very
small amount of data that covers all paths the code can take (or most paths
anyway).

https://github.com/holdenk/spark-testing-base

For integration testing we use automated routines to ensure that aggregate
values match an aggregate baseline.

Ryan

Ryan Adams
radams217@gmail.com

On Tue, Jun 12, 2018 at 11:51 AM, Lars Albertsson <lalle@mapflat.com> wrote:

> Hi,
>
> I wrote this answer to the same question a couple of years ago:
> https://www.mail-archive.com/user%40spark.apache.org/msg48032.html
>
> I have made a couple of presentations on the subject. Slides and video
> are linked on this page: http://www.mapflat.com/presentations/
>
> You can find more material in this list of resources:
> http://www.mapflat.com/lands/resources/reading-list
>
> Happy testing!
>
> Regards,
>
>
>
> Lars Albertsson
> Data engineering consultant
> www.mapflat.com
> https://twitter.com/lalleal
> +46 70 7687109
> Calendar: http://www.mapflat.com/calendar
>
>
> On Mon, May 21, 2018 at 2:24 PM, Steve Pruitt <bpruitt@opentext.com>
> wrote:
> > Hi,
> >
> >
> >
> > Can anyone recommend testing frameworks suitable for Spark jobs.
> Something
> > that can be integrated into a CI tool would be great.
> >
> >
> >
> > Thanks.
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Mime
View raw message