spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aditya Varun Chadha <>
Subject Embedding Spark Masters+Zk, Workers, SparkContext, App in single JVM, clustered (sorta for symmetric deployment)
Date Thu, 30 Oct 2014 07:07:38 GMT

Is it possible to start a spark standalone master inside my own JVM?

What I would like to do is the following:-
in my own main (object MyApp extends App):-
* Start zookeeper in embedded (and clustered) mode
* Start a spark master in same jvm referring to the above zookeeper quorum
for the HA (standby) behaviour
* Start spark worker in same jvm managed by the spark master quorum started
* Create a spark context against the above embedded master quorum in the
same jvm

I would then want to deploy many such "all in one" apps across a cluster
injecting appropriate (hopefully symmetric) configuration.

I understand that adding a new "node" like the the above to an existing
cluster would probably require some global reconfiguration (mostly because
of a new Zk and Master being added).

*An alternative* might be that I start spark masters and zk instances
externally, but I'd still want the the workers and the SparkContext to be
in a single jvm, that way I (can I?) can dynamically add workers (and
SparkContexts) to the same "spark cluster" without master reconfiguration.

job submission would happen via any of the open SparkContexts, possibly
coordinated through something like hazelcast/zookeeper

Pointers to sample code would be very helpful.

A debate around Pros/Cons of taking this approach would be really useful


Aditya Varun Chadha |

View raw message