spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <>
Subject Re: How to unit test spark streaming?
Date Tue, 07 Mar 2017 14:50:50 GMT
This depends on your target setup! I run for example for my open source libraries for spark
integration tests (a dedicated folder a side the unit tests) a local spark master, but also
use a minidfs cluster (to simulate HDFS on a node) and sometimes also a miniyarn cluster (see

 An example can be found here:

or - if you need Scala -

In both cases it is in the integration-tests (Java) or it (Scala) folder.

Spark Streaming - I have no open source example at hand, but basically you need to simulate
the source and the rest is as above.

 I will eventually write a blog post about this with more details.

> On 7 Mar 2017, at 13:04, kant kodali <> wrote:
> Hi All,
> How to unit test spark streaming or spark in general? How do I test the results of my
transformations? Also, more importantly don't we need to spawn master and worker JVM's either
in one or multiple nodes?
> Thanks!
> kant

To unsubscribe e-mail:

View raw message