spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anfernee Xu <anfernee...@gmail.com>
Subject Re: Question about yarn-cluster mode and spark.driver.allowMultipleContexts
Date Wed, 02 Dec 2015 05:43:50 GMT
Thanks Marcelo,

But I have a single server(JVM) that is creating SparkContext, are you
saying Spark supports multiple SparkContext in the same JVM? Could you
please clarify on this?

Thanks

Anfernee

On Tue, Dec 1, 2015 at 8:14 PM, Marcelo Vanzin <vanzin@cloudera.com> wrote:

> On Tue, Dec 1, 2015 at 3:32 PM, Anfernee Xu <anfernee.xu@gmail.com> wrote:
> > I have a long running backend server where I will create a short-lived
> Spark
> > job in response to each user request, base on the fact that by default
> > multiple Spark Context cannot be created in the same JVM, looks like I
> have
> > 2 choices
> >
> > 2) run my jobs in yarn-cluster mode instead yarn-client
>
> There's nothing in yarn-client mode that prevents you from doing what
> you describe. If you write some server for users to submit jobs to, it
> should work whether you start the context in yarn-client or
> yarn-cluster mode. It just might be harder to find out where it's
> running if you do it in cluster mode.
>
> --
> Marcelo
>



-- 
--Anfernee

Mime
View raw message