spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ofer Eliassaf <ofer.elias...@gmail.com>
Subject Re: spark standalone with multiple workers gives a warning
Date Thu, 06 Oct 2016 18:06:47 GMT
The slaves should connect to the master using the scripts in sbin...
You can read about it here:
http://spark.apache.org/docs/latest/spark-standalone.html

On Thu, Oct 6, 2016 at 6:46 PM, Mendelson, Assaf <Assaf.Mendelson@rsa.com>
wrote:

> Hi,
>
> I have a spark standalone cluster. On it, I am using 3 workers per node.
>
> So I added SPARK_WORKER_INSTANCES set to 3 in spark-env.sh
>
> The problem is, that when I run spark-shell I get the following warning:
>
> WARN SparkConf:
>
> SPARK_WORKER_INSTANCES was detected (set to '3').
>
> This is deprecated in Spark 1.0+.
>
>
>
> Please instead use:
>
> - ./spark-submit with --num-executors to specify the number of executors
>
> - Or set SPARK_EXECUTOR_INSTANCES
>
> - spark.executor.instances to configure the number of instances in the
> spark config.
>
>
>
> So how would I start a cluster of 3? SPARK_WORKER_INSTANCES is the only
> way I see to start the standalone cluster and the only way I see to define
> it is in spark-env.sh. The spark submit option, SPARK_EXECUTOR_INSTANCES
> and spark.executor.instances are all related to submitting the job.
>
>
>
> Any ideas?
>
> Thanks
>
>                 Assaf
>



-- 
Regards,
Ofer Eliassaf

Mime
View raw message