It's not possible to specify YARN RM paramers at command line of spark-submit time. You have to specify all resources that are available on your cluster to YARN upfront. If you want to limit amount of resource available for your Spark job, consider using YARN dynamic resource pools instead

http://www.cloudera.com/content/cloudera/en/documentation/cloudera-manager/v5-1-x/Cloudera-Manager-Managing-Clusters/cm5mc_resource_pools.html




--
Ruslan Dautkhanov

On Thu, Jul 2, 2015 at 4:20 PM, Alexander Waldin <awaldin@inflection.com> wrote:
Hi,

I'd like to specify the total sum of cores / memory as command line arguments with spark-submit. That is, I'd like to set yarn.nodemanager.resource.memory-mb and the yarn.nodemanager.resource.cpu-vcores parameters as described in this blog post.

when submitting through the command line, what is the correct way to do it? Is it:

--conf spark.yarn.nodemanager.resource.memory-mb=54g
--conf spark.yarn.nodemanager.resource.cpu-vcores=31

or

--conf yarn.nodemanager.resource.memory-mb=54g
--conf yarn.nodemanager.resource.cpu-vcores=31


or something else? I tried these, and I tried looking in the ResourceManager UI to see if they were set, but couldn't find them.

Thanks!

Alexander