spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <>
Subject Re: Question about yarn-cluster mode and spark.driver.allowMultipleContexts
Date Wed, 02 Dec 2015 04:14:43 GMT
On Tue, Dec 1, 2015 at 3:32 PM, Anfernee Xu <> wrote:
> I have a long running backend server where I will create a short-lived Spark
> job in response to each user request, base on the fact that by default
> multiple Spark Context cannot be created in the same JVM, looks like I have
> 2 choices
> 2) run my jobs in yarn-cluster mode instead yarn-client

There's nothing in yarn-client mode that prevents you from doing what
you describe. If you write some server for users to submit jobs to, it
should work whether you start the context in yarn-client or
yarn-cluster mode. It just might be harder to find out where it's
running if you do it in cluster mode.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message