spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Question about yarn-cluster mode and spark.driver.allowMultipleContexts
Date Wed, 02 Dec 2015 00:48:31 GMT
For #1, looks like the config is used in test suites:

        .set("spark.driver.allowMultipleContexts", "true")
./sql/core/src/test/scala/org/apache/spark/sql/MultiSQLContextsSuite.scala
        .set("spark.driver.allowMultipleContexts", "true")
./sql/core/src/test/scala/org/apache/spark/sql/execution/ExchangeCoordinatorSuite.scala
      .set("spark.driver.allowMultipleContexts", "false")
    val conf = new SparkConf().set("spark.driver.allowMultipleContexts",
"false")
        .set("spark.driver.allowMultipleContexts", "true")
        .set("spark.driver.allowMultipleContexts", "true"))
./core/src/test/scala/org/apache/spark/SparkContextSuite.scala

FYI

On Tue, Dec 1, 2015 at 3:32 PM, Anfernee Xu <anfernee.xu@gmail.com> wrote:

> Hi,
>
> I have a doubt regarding yarn-cluster mode and spark.driver.
> allowMultipleContexts for below usercases.
>
> I have a long running backend server where I will create a short-lived
> Spark job in response to each user request, base on the fact that by
> default multiple Spark Context cannot be created in the same JVM, looks
> like I have 2 choices
>
> 1) enable spark.driver.allowMultipleContexts
>
> 2) run my jobs in yarn-cluster mode instead yarn-client
>
> For 1) I cannot find any official document, so looks like it's not
> encouraged, isn't it?
> For 2), I want to make sure yarn-cluster will NOT hit such
> limitation(single SparkContext per VM), apparently I have to something in
> driver side to push the result set back to my application.
>
> Thanks
>
> --
> --Anfernee
>

Mime
View raw message