spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hichame El Khalfi <hich...@elkhalfi.com>
Subject Re: testing frameworks
Date Sun, 03 Feb 2019 21:58:54 GMT
Hi,
You can use pysparkling => https://github.com/svenkreiss/pysparkling
This lib is useful in case you have RDD.

Hope this helps,

Hichame

From: mmistroni@gmail.com
Sent: February 3, 2019 4:42 PM
To: radams217@gmail.com
Cc: lalle@mapflat.com; bpruitt@opentext.com; user@spark.apache.org
Subject: Re: testing frameworks


Hi
 sorry to resurrect this thread
Any spark libraries for testing code in pyspark?  the github code above seems related to Scala
following links in the original threads (and also LMGFY) i found out
<https://pypi.org/project/pytest-spark/>
pytest-spark ยท PyPI


w/kindest regards
 Marco




On Tue, Jun 12, 2018 at 6:44 PM Ryan Adams <radams217@gmail.com<mailto:radams217@gmail.com>>
wrote:
We use spark testing base for unit testing.  These tests execute on a very small amount of
data that covers all paths the code can take (or most paths anyway).

https://github.com/holdenk/spark-testing-base

For integration testing we use automated routines to ensure that aggregate values match an
aggregate baseline.

Ryan

Ryan Adams
radams217@gmail.com<mailto:radams217@gmail.com>

On Tue, Jun 12, 2018 at 11:51 AM, Lars Albertsson <lalle@mapflat.com<mailto:lalle@mapflat.com>>
wrote:
Hi,

I wrote this answer to the same question a couple of years ago:
https://www.mail-archive.com/user%40spark.apache.org/msg48032.html

I have made a couple of presentations on the subject. Slides and video
are linked on this page: http://www.mapflat.com/presentations/

You can find more material in this list of resources:
http://www.mapflat.com/lands/resources/reading-list

Happy testing!

Regards,



Lars Albertsson
Data engineering consultant
www.mapflat.com<http://www.mapflat.com>
https://twitter.com/lalleal
+46 70 7687109
Calendar: http://www.mapflat.com/calendar


On Mon, May 21, 2018 at 2:24 PM, Steve Pruitt <bpruitt@opentext.com<mailto:bpruitt@opentext.com>>
wrote:
> Hi,
>
>
>
> Can anyone recommend testing frameworks suitable for Spark jobs.  Something
> that can be integrated into a CI tool would be great.
>
>
>
> Thanks.
>
>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org<mailto:user-unsubscribe@spark.apache.org>



Mime
View raw message