spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dave DeCaprio (JIRA)" <>
Subject [jira] [Created] (SPARK-26988) Spark overwrites spark.scheduler.pool if set in configs
Date Mon, 25 Feb 2019 22:46:00 GMT
Dave DeCaprio created SPARK-26988:

             Summary: Spark overwrites spark.scheduler.pool if set in configs
                 Key: SPARK-26988
             Project: Spark
          Issue Type: Improvement
          Components: Scheduler
    Affects Versions: 2.4.0
            Reporter: Dave DeCaprio

If you set a default spark.scheduler.pool in your configuration when you create a SparkSession
and then you attempt to override that configuration by calling setLocalProperty on a SparkSession,
as described in the Spark documentation - [] -
it won't work.

Spark will go with the original pool name.

I've traced this down to SQLExecution.withSQLConfPropagated, which copies any key that starts
with "spark" from the the session state to the local properties.  The can end up overwriting
the scheduler, which is set by spark.scheduler.pool

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message