spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David McWhorter (Jira)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-22256) Introduce spark.mesos.driver.memoryOverhead
Date Tue, 08 Oct 2019 20:54:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947177#comment-16947177
] 

David McWhorter commented on SPARK-22256:
-----------------------------------------

I created this pull request https://github.com/pmackles/spark/pull/1 to update the branch
for this pull request with changes on branch https://github.com/dmcwhorter/spark/tree/dmcwhorter-SPARK-22256.

The updates do the following:
1. Rebase pmackles/paul-SPARK-22256 onto the latest apache/spark master branch
2. Add a test case for when the default value of spark.mesos.driver.memoryOverhead is 10%
of driver memory as requested in apache#21006

My hope is this update will make this pull request ready to merge and include in the next
spark release.

> Introduce spark.mesos.driver.memoryOverhead 
> --------------------------------------------
>
>                 Key: SPARK-22256
>                 URL: https://issues.apache.org/jira/browse/SPARK-22256
>             Project: Spark
>          Issue Type: Improvement
>          Components: Mesos
>    Affects Versions: 2.2.0, 2.3.0
>            Reporter: Cosmin Lehene
>            Priority: Minor
>              Labels: docker, memory, mesos
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> When running spark driver in a container such as when using the Mesos dispatcher service,
we need to apply the same rules as for executors in order to avoid the JVM going over the
allotted limit and then killed. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message