spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gourav Sengupta <>
Subject Re: Rest API for Spark2.3 submit on kubernetes(version 1.8.*) cluster
Date Thu, 22 Mar 2018 00:27:44 GMT
Hi Lucas,

Thanks a ton for responding.

have you used livy and SPARK in EMR? I am genuinely not sure how adding a
spark-submit in EMR is hard, it is just one line of code.

I must be missing something here

Gourav Sengupta

On Wed, Mar 21, 2018 at 2:37 PM, <>

> Speaking from experience, if you're already operating a kubernetes
> cluster.  Getting a spark workload operating there is nearly an order of
> magnitude simpler than working with / around EMR.
> That's not say EMR is excessively hard, just that Kubernetes is easier,
> all the steps to getting your application deployed are well documented and
> ultimately the whole process is more visible.
> Also, thanks for the link Yinan!  I'll be investigating that project!
> We have in the past used EMR for larger workloads and as soon as we
> announced our users could run those workloads on our k8s cluster everyone
> immediately moved their workloads over.  This was despite having Spark on
> K8s still in an infant state.  No one has expressed interest in moving back.
> G
> On 21 March 2018 at 07:32, Gourav Sengupta <>
> wrote:
>> Hi,
>> just out of curiosity, but since it in AWS, is there any specific reason
>> not to use EMR? Or any particular reason to use Kubernetes?
>> Regards,
>> Gourav Sengupta
>> On Wed, Mar 21, 2018 at 2:47 AM, purna pradeep <>
>> wrote:
>>> Im using kubernetes cluster on AWS to run spark jobs ,im using spark 2.3
>>> ,now i want to run spark-submit from AWS lambda function to k8s
>>> master,would like to know if there is any REST interface to run Spark
>>> submit on k8s Master

View raw message