spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anfernee Xu <anfernee...@gmail.com>
Subject Re: Question about yarn-cluster mode and spark.driver.allowMultipleContexts
Date Wed, 02 Dec 2015 00:51:45 GMT
Thanks Ted, so 1) is off from the table, can I go with 2), yarn-cluster
mode? As the driver is running as a Yarn container, it's should be OK for
my usercase, isn't it?

Anfernee

On Tue, Dec 1, 2015 at 4:48 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> For #1, looks like the config is used in test suites:
>
>         .set("spark.driver.allowMultipleContexts", "true")
> ./sql/core/src/test/scala/org/apache/spark/sql/MultiSQLContextsSuite.scala
>         .set("spark.driver.allowMultipleContexts", "true")
>
> ./sql/core/src/test/scala/org/apache/spark/sql/execution/ExchangeCoordinatorSuite.scala
>       .set("spark.driver.allowMultipleContexts", "false")
>     val conf = new SparkConf().set("spark.driver.allowMultipleContexts",
> "false")
>         .set("spark.driver.allowMultipleContexts", "true")
>         .set("spark.driver.allowMultipleContexts", "true"))
> ./core/src/test/scala/org/apache/spark/SparkContextSuite.scala
>
> FYI
>
> On Tue, Dec 1, 2015 at 3:32 PM, Anfernee Xu <anfernee.xu@gmail.com> wrote:
>
>> Hi,
>>
>> I have a doubt regarding yarn-cluster mode and spark.driver.
>> allowMultipleContexts for below usercases.
>>
>> I have a long running backend server where I will create a short-lived
>> Spark job in response to each user request, base on the fact that by
>> default multiple Spark Context cannot be created in the same JVM, looks
>> like I have 2 choices
>>
>> 1) enable spark.driver.allowMultipleContexts
>>
>> 2) run my jobs in yarn-cluster mode instead yarn-client
>>
>> For 1) I cannot find any official document, so looks like it's not
>> encouraged, isn't it?
>> For 2), I want to make sure yarn-cluster will NOT hit such
>> limitation(single SparkContext per VM), apparently I have to something in
>> driver side to push the result set back to my application.
>>
>> Thanks
>>
>> --
>> --Anfernee
>>
>
>


-- 
--Anfernee

Mime
View raw message