spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Commented] (SPARK-5349) Multiple spark shells should be able to share resources
Date Thu, 22 Jan 2015 10:52:34 GMT


Sean Owen commented on SPARK-5349:

I think this is a subset of which has been
implemented. This is a means to scale up and down the number of executors. It is for YARN,
but, I think this issue isn't acute in standalone mode, which is generally not multi-user.
The issue is not special to the shell.

> Multiple spark shells should be able to share resources
> -------------------------------------------------------
>                 Key: SPARK-5349
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 1.2.0
>            Reporter: Tobias Bertelsen
> The resource requirements of an interactive shell varies heavily. Sometimes heavy commands
are executed, and sometimes the user is thinking, getting coffee, interrupted etc... 
> A spark shell allocates a fixed number of worker cores (at least in standalone mode).
A user thus has the choice to either block other users from the cluster by allocating all
cores (default behavior), or restrict him/herself to only a few cores using the option {{--total-executor-cores}}.
Either way the cores allocated to the shell has low utilization, since they will be waiting
for the user a lot.
> Instead the spark shell allocate resources directly required to run the driver, and request
worker cores only when computation is performed on the RDDs.
> This should allow for multiple users, to use an interactive shell concurrently while
stille utilizing the entire cluster, when performing heavy operations.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message