spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ruslan Dautkhanov <dautkha...@gmail.com>
Subject Re: configuring max sum of cores and memory in cluster through command line
Date Sun, 05 Jul 2015 17:09:43 GMT
It's not possible to specify YARN RM paramers at command line of
spark-submit time. You have to specify all resources that are available on
your cluster to YARN upfront. If you want to limit amount of resource
available for your Spark job, consider using YARN dynamic resource pools
instead

http://www.cloudera.com/content/cloudera/en/documentation/cloudera-manager/v5-1-x/Cloudera-Manager-Managing-Clusters/cm5mc_resource_pools.html




-- 
Ruslan Dautkhanov

On Thu, Jul 2, 2015 at 4:20 PM, Alexander Waldin <awaldin@inflection.com>
wrote:

>  Hi,
>
> I'd like to specify the total sum of cores / memory as command line
> arguments with spark-submit. That is, I'd like to set
> yarn.nodemanager.resource.memory-mb and the
> yarn.nodemanager.resource.cpu-vcores parameters as described in this blog
> <http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/>
> post.
>
> when submitting through the command line, what is the correct way to do
> it? Is it:
>
> --conf spark.yarn.nodemanager.resource.memory-mb=54g
> --conf spark.yarn.nodemanager.resource.cpu-vcores=31
>
> or
>
> --conf yarn.nodemanager.resource.memory-mb=54g
> --conf yarn.nodemanager.resource.cpu-vcores=31
>
>
> or something else? I tried these, and I tried looking in the
> ResourceManager UI to see if they were set, but couldn't find them.
>
> Thanks!
>
> Alexander
>

Mime
View raw message