spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin Loncaric (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-26192) MesosClusterScheduler reads options from dispatcher conf instead of submission conf
Date Thu, 28 Feb 2019 19:15:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-26192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Martin Loncaric updated SPARK-26192:
------------------------------------
    Description: 
There is at least one option accessed in MesosClusterScheduler that should come from the submission's
configuration instead of the dispatcher's:

spark.mesos.fetcherCache.enable

Coincidentally, the spark.mesos.fetcherCache.enable option was previously misnamed, as referenced
in the linked JIRA.

  was:
There are at least two options accessed in MesosClusterScheduler that should come from the
submission's configuration instead of the dispatcher's:

spark.app.name
spark.mesos.fetcherCache.enable

This means that all Mesos tasks for Spark drivers have uninformative names of the form "Driver
for (MainClass)" rather than the configured application name, and Spark drivers never cache
files unless the caching setting is specified on dispatcher as well. Coincidentally, the spark.mesos.fetchCache.enable
option was misnamed, as referenced in the linked JIRA.


> MesosClusterScheduler reads options from dispatcher conf instead of submission conf
> -----------------------------------------------------------------------------------
>
>                 Key: SPARK-26192
>                 URL: https://issues.apache.org/jira/browse/SPARK-26192
>             Project: Spark
>          Issue Type: Bug
>          Components: Mesos
>    Affects Versions: 2.3.0, 2.3.1, 2.3.2, 2.4.0
>            Reporter: Martin Loncaric
>            Priority: Major
>
> There is at least one option accessed in MesosClusterScheduler that should come from
the submission's configuration instead of the dispatcher's:
> spark.mesos.fetcherCache.enable
> Coincidentally, the spark.mesos.fetcherCache.enable option was previously misnamed, as
referenced in the linked JIRA.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message