Hmm, so when I submit an application with `spark-submit`, I need to guarantee it resources using YARN queues and not Spark's scheduler pools. Is that correct?

When are Spark's scheduler pools relevant/useful in this context?

On Wed, Apr 5, 2017 at 3:54 PM Mark Hamstra <mark@clearstorydata.com> wrote:
grrr... s/your/you're/

On Wed, Apr 5, 2017 at 12:54 PM, Mark Hamstra <mark@clearstorydata.com> wrote:
Your mixing up different levels of scheduling. Spark's fair scheduler pools are about scheduling Jobs, not Applications; whereas YARN queues with Spark are about scheduling Applications, not Jobs. 

On Wed, Apr 5, 2017 at 12:27 PM, Nick Chammas <nicholas.chammas@gmail.com> wrote:
I'm having trouble understanding the difference between Spark fair scheduler pools and YARN queues. Do they conflict? Does one override the other?

I posted a more detailed question about an issue I'm having with this on Stack Overflow: http://stackoverflow.com/q/43239921/877069

Nick



View this message in context: Spark fair scheduler pools vs. YARN queues
Sent from the Apache Spark User List mailing list archive at Nabble.com.