spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ajay Chander <>
Subject Re: submit_spark_job_to_YARN
Date Sun, 30 Aug 2015 18:54:40 GMT
Hi David,

Thanks for responding! My main intention was to submit spark Job/jar to
yarn cluster from my eclipse with in the code. Is there any way that I
could pass my yarn configuration somewhere in the code to submit the jar to
the cluster?

Thank you,

On Sunday, August 30, 2015, David Mitchell <> wrote:

> Hi Ajay,
> Are you trying to save to your local file system or to HDFS?
> // This would save to HDFS under "/user/hadoop/counter"
> counter.saveAsTextFile("/user/hadoop/counter");
> David
> On Sun, Aug 30, 2015 at 11:21 AM, Ajay Chander <
> <javascript:_e(%7B%7D,'cvml','');>> wrote:
>> Hi Everyone,
>> Recently we have installed spark on yarn in hortonworks cluster. Now I am
>> trying to run a wordcount program in my eclipse and I
>> did setMaster("local") and I see the results that's as expected. Now I want
>> to submit the same job to my yarn cluster from my eclipse. In storm
>> basically I was doing the same by using StormSubmitter class and by passing
>> nimbus & zookeeper host to Config object. I was looking for something
>> exactly the same.
>> When I went through the documentation online, it read that I am suppose
>> to "export HADOOP_HOME_DIR=path to the conf dir". So now I copied the conf
>> folder from one of sparks gateway node to my local Unix box. Now I did
>> export that dir...
>> export HADOOP_HOME_DIR=/Users/user1/Documents/conf/
>> And I did the same in .bash_profile too. Now when I do echo
>> $HADOOP_HOME_DIR, I see the path getting printed in the command prompt. Now
>> my assumption is, in my program when I change setMaster("local") to
>> setMaster("yarn-client") my program should pick up the resource mangers i.e
>> yarn cluster info from the directory which I have exported and the job
>> should get submitted to resolve manager from my eclipse. But somehow it's
>> not happening. Please tell me if my assumption is wrong or if I am missing
>> anything here.
>> I have attached the word count program that I was using. Any help is
>> highly appreciated.
>> Thank you,
>> Ajay
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> <javascript:_e(%7B%7D,'cvml','');>
>> For additional commands, e-mail:
>> <javascript:_e(%7B%7D,'cvml','');>
> --
> ### Confidential e-mail, for recipient's (or recipients') eyes only, not
> for distribution. ###

View raw message