spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Cheah <>
Subject Re: k8s orchestrating Spark service
Date Mon, 01 Jul 2019 22:26:14 GMT
I would recommend looking into Spark’s native support for running on Kubernetes. One can
just start the application against Kubernetes directly using spark-submit in cluster mode
or starting the Spark context with the right parameters in client mode. See


I would think that building Helm around this architecture of running Spark applications would
be easier than running a Spark standalone cluster. But admittedly I’m not very familiar
with the Helm technology – we just use spark-submit.


-Matt Cheah

From: Pat Ferrel <>
Date: Sunday, June 30, 2019 at 12:55 PM
To: "" <>
Subject: k8s orchestrating Spark service


We're trying to setup a system that includes Spark. The rest of the services have good Docker
containers and Helm charts to start from.


Spark on the other hand is proving difficult. We forked a container and have tried to create
our own chart but are having several problems with this.


So back to the community… Can anyone recommend a Docker Container + Helm Chart for use with
Kubernetes to orchestrate:
Spark standalone Master
several Spark Workers/Executors
This not a request to use k8s to orchestrate Spark Jobs, but the service cluster itself.




View raw message