spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lars Albertsson <la...@mapflat.com>
Subject Re: Testing Apache Spark applications
Date Thu, 15 Nov 2018 20:19:13 GMT
My previous answers to this question can be found in the archives, along
with some other responses:

http://apache-spark-user-list.1001560.n3.nabble.com/testing-frameworks-td32251.html
https://www.mail-archive.com/user%40spark.apache.org/msg48032.html

I have made a couple of presentations on the subject. Slides and video
are linked on this page: http://www.mapflat.com/presentations/

You can find more material in this list of resources:
http://www.mapflat.com/lands/resources/reading-list

Happy testing!

Regards,


Lars Albertsson
Data engineering entrepreneur
www.mimeria.com, www.mapflat.com
https://twitter.com/lalleal
+46 70 7687109


On Thu, Nov 15, 2018 at 6:45 PM <Omer.Ozsakarya@sony.com> wrote:

> Hi all,
>
>
>
> How are you testing your Spark applications?
>
> We are writing features by using Cucumber. This is testing the behaviours.
> Is this called functional test or integration test?
>
>
>
> We are also planning to write unit tests.
>
>
>
> For instance we have a class like below. It has one method. This methos is
> implementing several things: like DataFrame operations, saving DataFrame
> into database table, insert, update,delete statements.
>
>
>
> Our classes generally contains 2 or 3 methods. These methods cover a lot
> of tasks in the same function defintion. (like the function below)
>
> So I am not sure how I can write unit tests for these classes and methods.
>
> Do you have any suggestion?
>
>
>
> class CustomerOperations
>
>
>
>    def doJob(inputDataFrame : DataFrame) = {
>
>            // definitions (value/variable)
>
>            // spark context, session etc definition
>
>
>
>           //  filtering, cleansing on inputDataframe and save results on a
> new dataframe
>
>           // insert new dataframe to a database table
>
>          //  several insert/update/delete statements on the database tables
>
>
>
>     }
>
>
>
>
>

Mime
View raw message