spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lars Albertsson <>
Subject Re: Integration testing Framework Spark SQL Scala
Date Mon, 02 Nov 2020 13:09:34 GMT

Sorry for the very slow reply - I am far behind in my mailing list

You'll find a few slides covering the topic in this presentation:

Video here:


Lars Albertsson
Data engineering entrepreneur,
+46 70 7687109

On Tue, Feb 25, 2020 at 7:46 PM Ruijing Li <> wrote:
> Just wanted to follow up on this. If anyone has any advice, I’d be interested in learning
> On Thu, Feb 20, 2020 at 6:09 PM Ruijing Li <> wrote:
>> Hi all,
>> I’m interested in hearing the community’s thoughts on best practices to do integration
testing for spark sql jobs. We run a lot of our jobs with cloud infrastructure and hdfs -
this makes debugging a challenge for us, especially with problems that don’t occur from
just initializing a sparksession locally or testing with spark-shell. Ideally, we’d like
some sort of docker container emulating hdfs and spark cluster mode, that you can run locally.
>> Any test framework, tips, or examples people can share? Thanks!
>> --
>> Cheers,
>> Ruijing Li
> --
> Cheers,
> Ruijing Li

To unsubscribe e-mail:

View raw message