Hey Gary,

The answer to both of your questions is that much of it is up to the application.

For (1), the standalone master can set "spark.deploy.defaultCores" to limit the number of cores each application can grab. However, the application can override this with the applications-specific "spark.cores.max", meaning there is currently nothing the master can do if the application is greedy and demands all the cores in the world.

For (2), I am not aware of an existing way the standalone master can kill a user application. The most you can do is to go to the application SparkUI and kill the stages (there is a button), though this is not specific to standalone mode.

There is currently a lot of trust between the standalone master and the application. Maybe this is not always a good thing. :)

-Andrew


2014-08-06 12:23 GMT-07:00 Gary Malouf <malouf.gary@gmail.com>:
I have a few questions about managing Spark memory:

1) In a standalone setup, is their any cpu prioritization across users running jobs?  If so, what is the behavior here?

2) With Spark 1.1, users will more easily be able to run drivers/shells from remote locations that do not cause firewall headaches.  Is there a way to kill an individual user's job from the console without killing workers?  We are in Mesos and are not aware of an easy way to handle this, but I imagine standalone mode may handle this.