spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gourav Sengupta <gourav.sengu...@gmail.com>
Subject Re: Rest API for Spark2.3 submit on kubernetes(version 1.8.*) cluster
Date Thu, 22 Mar 2018 00:27:44 GMT
Hi Lucas,

Thanks a ton for responding.

have you used livy and SPARK in EMR? I am genuinely not sure how adding a
spark-submit in EMR is hard, it is just one line of code.

I must be missing something here


Regards,
Gourav Sengupta

On Wed, Mar 21, 2018 at 2:37 PM, lucas.gary@gmail.com <lucas.gary@gmail.com>
wrote:

> Speaking from experience, if you're already operating a kubernetes
> cluster.  Getting a spark workload operating there is nearly an order of
> magnitude simpler than working with / around EMR.
>
> That's not say EMR is excessively hard, just that Kubernetes is easier,
> all the steps to getting your application deployed are well documented and
> ultimately the whole process is more visible.
>
> Also, thanks for the link Yinan!  I'll be investigating that project!
>
> We have in the past used EMR for larger workloads and as soon as we
> announced our users could run those workloads on our k8s cluster everyone
> immediately moved their workloads over.  This was despite having Spark on
> K8s still in an infant state.  No one has expressed interest in moving back.
>
> G
>
> On 21 March 2018 at 07:32, Gourav Sengupta <gourav.sengupta@gmail.com>
> wrote:
>
>> Hi,
>>
>> just out of curiosity, but since it in AWS, is there any specific reason
>> not to use EMR? Or any particular reason to use Kubernetes?
>>
>>
>> Regards,
>> Gourav Sengupta
>>
>> On Wed, Mar 21, 2018 at 2:47 AM, purna pradeep <purna2pradeep@gmail.com>
>> wrote:
>>
>>> Im using kubernetes cluster on AWS to run spark jobs ,im using spark 2.3
>>> ,now i want to run spark-submit from AWS lambda function to k8s
>>> master,would like to know if there is any REST interface to run Spark
>>> submit on k8s Master
>>
>>
>>
>

Mime
View raw message