spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Narrell <matt.narr...@gmail.com>
Subject Re: Execute Spark programs from local machine on Yarn-hadoop cluster
Date Mon, 24 Nov 2014 01:50:53 GMT
I think this IS possible?

You must set the HADOOP_CONF_DIR variable on the machine you’re running the Java process
that creates the SparkContext.  The Hadoop configuration specifies the YARN ResourceManager
IPs, and Spark will use that configuration.  

mn

> On Nov 21, 2014, at 8:10 AM, Prannoy <prannoy@sigmoidanalytics.com> wrote:
> 
> Hi naveen, 
> 
> I dont think this is possible. If you are setting the master with your cluster details
you cannot execute any job from your local machine. You have to execute the jobs inside your
yarn machine so that sparkconf is able to connect with all the provided details. 
> 
> If this is not the case such give a detail explaintation of what exactly you are trying
to do :)
> 
> Thanks.
> 
> On Fri, Nov 21, 2014 at 8:11 PM, Naveen Kumar Pokala [via Apache Spark User List] <[hidden
email] <x-msg://4/user/SendEmail.jtp?type=node&node=19484&i=0>> wrote:
> Hi,
> 
>  
> 
> I am executing my spark jobs on yarn cluster by forming conf object in the following
way.
> 
>  
> 
> SparkConf conf = new SparkConf().setAppName("NewJob").setMaster("yarn-cluster");
> 
>  
> 
> Now I want to execute spark jobs from my local machine how to do that.
> 
>  
> 
> What I mean is there a way to give IP address, port all the details to connect a master(YARN)
on some other network from my local spark Program.
> 
>  
> 
> -Naveen
> 
> 
> 
> If you reply to this email, your message will be added to the discussion below:
> http://apache-spark-user-list.1001560.n3.nabble.com/Execute-Spark-programs-from-local-machine-on-Yarn-hadoop-cluster-tp19482.html
<http://apache-spark-user-list.1001560.n3.nabble.com/Execute-Spark-programs-from-local-machine-on-Yarn-hadoop-cluster-tp19482.html>
> To start a new topic under Apache Spark User List, email [hidden email] <x-msg://4/user/SendEmail.jtp?type=node&node=19484&i=1>

> To unsubscribe from Apache Spark User List, click here <applewebdata://63F90BD1-F0D8-41D0-A1DE-B7ACE896B9D9>.
> NAML <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> 
> View this message in context: Re: Execute Spark programs from local machine on Yarn-hadoop
cluster <http://apache-spark-user-list.1001560.n3.nabble.com/Execute-Spark-programs-from-local-machine-on-Yarn-hadoop-cluster-tp19482p19484.html>
> Sent from the Apache Spark User List mailing list archive <http://apache-spark-user-list.1001560.n3.nabble.com/>
at Nabble.com.


Mime
View raw message