spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Cheah <mch...@palantir.com>
Subject Re: k8s orchestrating Spark service
Date Mon, 01 Jul 2019 22:26:14 GMT
I would recommend looking into Spark’s native support for running on Kubernetes. One can
just start the application against Kubernetes directly using spark-submit in cluster mode
or starting the Spark context with the right parameters in client mode. See https://spark.apache.org/docs/latest/running-on-kubernetes.html

 

I would think that building Helm around this architecture of running Spark applications would
be easier than running a Spark standalone cluster. But admittedly I’m not very familiar
with the Helm technology – we just use spark-submit.

 

-Matt Cheah

From: Pat Ferrel <pat@occamsmachete.com>
Date: Sunday, June 30, 2019 at 12:55 PM
To: "user@spark.apache.org" <user@spark.apache.org>
Subject: k8s orchestrating Spark service

 

We're trying to setup a system that includes Spark. The rest of the services have good Docker
containers and Helm charts to start from.

 

Spark on the other hand is proving difficult. We forked a container and have tried to create
our own chart but are having several problems with this.

 

So back to the community… Can anyone recommend a Docker Container + Helm Chart for use with
Kubernetes to orchestrate:
Spark standalone Master
several Spark Workers/Executors
This not a request to use k8s to orchestrate Spark Jobs, but the service cluster itself.

 

Thanks

 


Mime
View raw message