This website explains it very clear, if you are using Yarn.


https://www.cloudera.com/documentation/enterprise/5-6-x/topics/cdh_ig_running_spark_on_yarn.html

www.cloudera.com
When Spark applications run on a YARN cluster manager, resource management, scheduling, and security are controlled by YARN.





From: Adaryl Wakefield <adaryl.wakefield@hotmail.com>
Sent: Tuesday, March 7, 2017 8:53 PM
To: Koert Kuipers
Cc: user@spark.apache.org
Subject: RE: finding Spark Master
 

Ah so I see setMaster(Ďyarn-clientí). Hmm.

 

What I was ultimately trying to do was develop with Eclipse on my windows box and have the code point to my cluster so it executes there instead of my local windows machine. Perhaps Iím going about this wrong.

 

Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics, LLC
913.938.6685

www.massstreet.net

www.linkedin.com/in/bobwakefieldmba
Twitter: @BobLovesData

 

From: Koert Kuipers [mailto:koert@tresata.com]
Sent: Tuesday, March 7, 2017 7:47 PM
To: Adaryl Wakefield <adaryl.wakefield@hotmail.com>
Cc: user@spark.apache.org
Subject: Re: finding Spark Master

 

assuming this is running on yarn there is really spark-master. every job created its own "master" within a yarn application.

 

On Tue, Mar 7, 2017 at 6:27 PM, Adaryl Wakefield <adaryl.wakefield@hotmail.com> wrote:

Iím running a three node cluster along with Spark along with Hadoop as part of a HDP stack. How do I find my Spark Master? Iím just seeing the clients. Iím trying to figure out what goes in setMaster() aside from local[*].

 

Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics, LLC
913.938.6685

www.massstreet.net

www.linkedin.com/in/bobwakefieldmba
Twitter: @BobLovesData