Ted, Mich,

Thanks for your replies. I ended up using sparkConf.set(<cores>); and accepted cores as a parameter. But still not sure why spark-submits's executor-cores or driver-cores property did not work. setting cores within main method seems to be bit cumbersome .

Thanks again,
Shridhar


On Wed, Mar 30, 2016 at 8:42 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:
Hi Ted

Can specify the core as follows for example 12 cores?:

  val conf = new SparkConf().
               setAppName("ImportStat").
               setMaster("local[12]").
               set("spark.driver.allowMultipleContexts", "true").
               set("spark.hadoop.validateOutputSpecs", "false")
  val sc = new SparkContext(conf)
 


On 30 March 2016 at 14:59, Ted Yu <yuzhihong@gmail.com> wrote:
-c CORES--cores CORESTotal CPU cores to allow Spark applications to use on the machine (default: all available); only on worker

bq. sc.getConf().set()

I think you should use this pattern (shown in https://spark.apache.org/docs/latest/spark-standalone.html):

val conf = new SparkConf()
             .setMaster(...)
             .setAppName(...)
             .set("spark.cores.max", "1")
val sc = new SparkContext(conf)

On Wed, Mar 30, 2016 at 5:46 AM, vetal king <greenvetal@gmail.com> wrote:
Hi all,

While submitting Spark Job I am am specifying options --executor-cores 1  and --driver-cores 1. However, when the job was submitted, the job used all available cores. So I tried to limit the cores within my main function             sc.getConf().set("spark.cores.max", "1"); however it still used all available cores

I am using Spark in standalone mode (spark://<hostname>:7077)

Any idea what I am missing?
Thanks in Advance,

Shridhar