spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From شجاع الرحمن بیگ <shujamug...@gmail.com>
Subject Setting Spark Worker Memory
Date Wed, 11 May 2016 15:38:07 GMT
Hi All,

I need to set same memory and core for each worker on same machine and for
this purpose, I have set the following properties in conf/spark-env.sh

export SPARK_EXECUTOR_INSTANCE=3
export SPARK_WORKER_CORES=3
export SPARK_WORKER_MEMORY=8g

but only one worker is getting desired memory and cores and other two are
not. here is the log of master.

...
6/05/11 17:04:40 INFO Master: I have been elected leader! New state: ALIVE
16/05/11 17:04:43 INFO Master: Registering worker 11.14.224.24:53923 with 3
cores, 8.0 GB RAM
16/05/11 17:04:49 INFO Master: Registering worker 11.14.224.24:55072 with 2
cores, 1020.7 GB RAM
16/05/11 17:05:07 INFO Master: Registering worker 11.14.224.24:49702 with 2
cores, 1020.7 GB RAM
...


Could you please let me know the solution

Thanks
Shuja

-- 
Regards
Shuja-ur-Rehman Baig
<http://pk.linkedin.com/in/shujamughal>

Mime
View raw message