spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geoff Von Allmen <ge...@ibleducation.com>
Subject Re: Fast Unit Tests
Date Tue, 01 May 2018 15:44:24 GMT
I am pretty new to spark/scala myself, but I just recently implemented unit
tests to test my transformations/aggregations and such myself.

I’m using the mrpowers spark-fast-tests
<https://github.com/MrPowers/spark-fast-tests> and spark-daria
<https://github.com/MrPowers/spark-daria> libraries.

I am also using a JDBC sink in the foreach writer. I’ve mocked the sink to
place the generated MYSQL statements in a global object and then I compare
the output there to an expected set of mysql statements.

I’m running this with FlatSpec ScalaTests, where my spark inputs are
manually generated fixtures for each test case. Everything seems to be
running well and it's nice and quick.
​

On Tue, May 1, 2018 at 8:25 AM, marcos rebelo <oleber@gmail.com> wrote:

> Hey all,
>
> We are using Scala and SQL heavily, but I have a problem with VERY SLOW
> Unit Tests.
>
> Is there a way to do fast Unit Tests on Spark?
>
> How are you guys going around it?
>
> Best Regards
> Marcos Rebelo
>

Mime
View raw message