spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Goldsborough <joshgoldsboroughs...@gmail.com>
Subject Re: Rest API for Spark2.3 submit on kubernetes(version 1.8.*) cluster
Date Wed, 21 Mar 2018 23:09:23 GMT
Purna,

It's a bit tangental to your original question but heads up that Amazon EKS
is in Preview right now:
https://aws.amazon.com/eks/

I don't know if it actually allows a nice interface between k8s hosted
Spark & Lamda functions (my suspicion is it won't fix your problem), but
might be something worth taking a peak at, since you're already doing AWS
hosted K8s stuff.

On Wed, Mar 21, 2018 at 9:37 AM, lucas.gary@gmail.com <lucas.gary@gmail.com>
wrote:

> Speaking from experience, if you're already operating a kubernetes
> cluster.  Getting a spark workload operating there is nearly an order of
> magnitude simpler than working with / around EMR.
>
> That's not say EMR is excessively hard, just that Kubernetes is easier,
> all the steps to getting your application deployed are well documented and
> ultimately the whole process is more visible.
>
> Also, thanks for the link Yinan!  I'll be investigating that project!
>
> We have in the past used EMR for larger workloads and as soon as we
> announced our users could run those workloads on our k8s cluster everyone
> immediately moved their workloads over.  This was despite having Spark on
> K8s still in an infant state.  No one has expressed interest in moving back.
>
> G
>
> On 21 March 2018 at 07:32, Gourav Sengupta <gourav.sengupta@gmail.com>
> wrote:
>
>> Hi,
>>
>> just out of curiosity, but since it in AWS, is there any specific reason
>> not to use EMR? Or any particular reason to use Kubernetes?
>>
>>
>> Regards,
>> Gourav Sengupta
>>
>> On Wed, Mar 21, 2018 at 2:47 AM, purna pradeep <purna2pradeep@gmail.com>
>> wrote:
>>
>>> Im using kubernetes cluster on AWS to run spark jobs ,im using spark 2.3
>>> ,now i want to run spark-submit from AWS lambda function to k8s
>>> master,would like to know if there is any REST interface to run Spark
>>> submit on k8s Master
>>
>>
>>
>

Mime
View raw message