spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Or <>
Subject Re: Spark memory management
Date Wed, 06 Aug 2014 21:18:58 GMT
Hey Gary,

The answer to both of your questions is that much of it is up to the

For (1), the standalone master can set "spark.deploy.defaultCores" to limit
the number of cores each application can grab. However, the application can
override this with the applications-specific "spark.cores.max", meaning
there is currently nothing the master can do if the application is greedy
and demands all the cores in the world.

For (2), I am not aware of an existing way the standalone master can kill a
user application. The most you can do is to go to the application SparkUI
and kill the stages (there is a button), though this is not specific to
standalone mode.

There is currently a lot of trust between the standalone master and the
application. Maybe this is not always a good thing. :)


2014-08-06 12:23 GMT-07:00 Gary Malouf <>:

> I have a few questions about managing Spark memory:
> 1) In a standalone setup, is their any cpu prioritization across users
> running jobs?  If so, what is the behavior here?
> 2) With Spark 1.1, users will more easily be able to run drivers/shells
> from remote locations that do not cause firewall headaches.  Is there a way
> to kill an individual user's job from the console without killing workers?
>  We are in Mesos and are not aware of an easy way to handle this, but I
> imagine standalone mode may handle this.

View raw message