spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <jornfra...@gmail.com>
Subject Re: How to unit test spark streaming?
Date Tue, 07 Mar 2017 14:50:50 GMT
This depends on your target setup! I run for example for my open source libraries for spark
integration tests (a dedicated folder a side the unit tests) a local spark master, but also
use a minidfs cluster (to simulate HDFS on a node) and sometimes also a miniyarn cluster (see
https://wiki.apache.org/hadoop/HowToDevelopUnitTests).

 An example can be found here:  https://github.com/ZuInnoTe/hadoopcryptoledger/tree/master/examples/spark-bitcoinblock


or - if you need Scala - 
https://github.com/ZuInnoTe/hadoopcryptoledger/tree/master/examples/scala-spark-bitcoinblock


In both cases it is in the integration-tests (Java) or it (Scala) folder.

Spark Streaming - I have no open source example at hand, but basically you need to simulate
the source and the rest is as above.

 I will eventually write a blog post about this with more details.

> On 7 Mar 2017, at 13:04, kant kodali <kanth909@gmail.com> wrote:
> 
> Hi All,
> 
> How to unit test spark streaming or spark in general? How do I test the results of my
transformations? Also, more importantly don't we need to spawn master and worker JVM's either
in one or multiple nodes?
> 
> Thanks!
> kant

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message