spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Mackles <>
Subject [mesos] dispatcher and mesos constraints applied to driver - SPARK-19606
Date Mon, 02 Oct 2017 22:56:00 GMT
Hi - I was hoping to revive the discussion regarding the application of mesos constraints to
driver tasks. I added some comments to: <>

Basically, I am proposing the introduction of an additional optional configuration property
named "spark.mesos.constraints.driver" which would function the same as "spark.mesos.constraints"
except that it would be applied to driver tasks by the mesos dispatcher.

Aside from being backwards compatible, it would allow driver tasks to be constrained to either
the same or different resources as the executors.

This would provide better support for multi-tenant cluster as well as provide a means for
limiting the number of drivers that are allowed to run concurrently on a cluster since too
many drivers can lead to starvation scenarios.

If folks agree that this is a good change, I am about 75% done with a PR to implement the
above. There is an alternative proposal (and PR) in the ticket as well if that is the preferred


View raw message