spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Goldsborough <>
Subject Re: Rest API for Spark2.3 submit on kubernetes(version 1.8.*) cluster
Date Wed, 21 Mar 2018 23:09:23 GMT

It's a bit tangental to your original question but heads up that Amazon EKS
is in Preview right now:

I don't know if it actually allows a nice interface between k8s hosted
Spark & Lamda functions (my suspicion is it won't fix your problem), but
might be something worth taking a peak at, since you're already doing AWS
hosted K8s stuff.

On Wed, Mar 21, 2018 at 9:37 AM, <>

> Speaking from experience, if you're already operating a kubernetes
> cluster.  Getting a spark workload operating there is nearly an order of
> magnitude simpler than working with / around EMR.
> That's not say EMR is excessively hard, just that Kubernetes is easier,
> all the steps to getting your application deployed are well documented and
> ultimately the whole process is more visible.
> Also, thanks for the link Yinan!  I'll be investigating that project!
> We have in the past used EMR for larger workloads and as soon as we
> announced our users could run those workloads on our k8s cluster everyone
> immediately moved their workloads over.  This was despite having Spark on
> K8s still in an infant state.  No one has expressed interest in moving back.
> G
> On 21 March 2018 at 07:32, Gourav Sengupta <>
> wrote:
>> Hi,
>> just out of curiosity, but since it in AWS, is there any specific reason
>> not to use EMR? Or any particular reason to use Kubernetes?
>> Regards,
>> Gourav Sengupta
>> On Wed, Mar 21, 2018 at 2:47 AM, purna pradeep <>
>> wrote:
>>> Im using kubernetes cluster on AWS to run spark jobs ,im using spark 2.3
>>> ,now i want to run spark-submit from AWS lambda function to k8s
>>> master,would like to know if there is any REST interface to run Spark
>>> submit on k8s Master

View raw message