spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rapelly kartheek <kartheek.m...@gmail.com>
Subject Re: rsync problem
Date Fri, 26 Sep 2014 15:43:08 GMT
Hi,

This is the command I am using for submitting my application, SimpleApp:

./bin/spark-submit --class org.apache.spark.examples.SimpleApp
--deploy-mode client --master spark://karthik:7077
$SPARK_HOME/examples/*/scala-*/spark-examples-*.jar /text-data


On Thu, Sep 25, 2014 at 6:52 AM, Tobias Pfeiffer <tgp@preferred.jp> wrote:

> Hi,
>
> I assume you unintentionally did not reply to the list, so I'm adding it
> back to CC.
>
> How do you submit your job to the cluster?
>
> Tobias
>
>
> On Thu, Sep 25, 2014 at 2:21 AM, rapelly kartheek <kartheek.mbms@gmail.com
> > wrote:
>
>> How do I find out whether a node in the cluster is a master or slave??
>> Till now I was thinking that slaves file under the conf folder makes the
>> difference. Also, the MASTER_MASTER_IP in the spark-env.sh file.
>>
>> what else differentiates a slave from the master??
>>
>> On Wed, Sep 24, 2014 at 10:46 PM, rapelly kartheek <
>> kartheek.mbms@gmail.com> wrote:
>>
>>> The job execution is taking place perfectly. Previously, all my print
>>> statements used to be stored in spark/work/*/stdout file. But, now after
>>> doing the rsync, I find that none of the prtint statements are getting
>>> reflected in the stdout file under work folder. But, when I go to the code,
>>> I find the statements in the code. But, they are not reflected into the
>>> stdout file as before.
>>>
>>> Can you please tell me where I went wrong.  All I want is to see my
>>> mofication in the code getting relected in output
>>> .
>>>
>>> On Wed, Sep 24, 2014 at 10:22 PM, rapelly kartheek <
>>> kartheek.mbms@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have a very important and fundamental doubt: I have rsynced the
>>>> entire spark folder from the master to all slaves in the cluster. When I
>>>> execute a job, its working perfectly. But, when I rsync the entire spark
>>>> folder of the master to all the slaves, is it not that I am sending the
>>>> master configurations to all the slaves and making the slaves behave like
>>>> master??
>>>>
>>>> First of all, is it correct to rsync the entire spark folder??
>>>> But, if I change only one file, then how do I rsync it to all??
>>>>
>>>> On Fri, Sep 19, 2014 at 8:44 PM, rapelly kartheek <
>>>> kartheek.mbms@gmail.com> wrote:
>>>>
>>>>> Thank you Soumya Simantha and Tobias. I've deleted the contents of the
>>>>> work folder in all the nodes.
>>>>> Now its working perfectly as it was before.
>>>>>
>>>>> Thank you
>>>>> Karthik
>>>>>
>>>>> On Fri, Sep 19, 2014 at 4:46 PM, Soumya Simanta <
>>>>> soumya.simanta@gmail.com> wrote:
>>>>>
>>>>>> One possible reason is maybe that the checkpointing directory
>>>>>> $SPARK_HOME/work is rsynced as well.
>>>>>> Try emptying the contents of the work folder on each node and try
>>>>>> again.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 19, 2014 at 4:53 AM, rapelly kartheek <
>>>>>> kartheek.mbms@gmail.com> wrote:
>>>>>>
>>>>>>> I
>>>>>>> * followed this command:rsync -avL --progress path/to/spark-1.0.0
>>>>>>> username@destinationhostname:*
>>>>>>>
>>>>>>>
>>>>>>> *path/to/destdirectory. Anyway, for now, I did it individually
for
>>>>>>> each node.*
>>>>>>>
>>>>>>> I have copied to each node at a time individually using the above
>>>>>>> command. So, I guess the copying may not contain any mixture
of files.
>>>>>>> Also, as of now, I am not facing any MethodNotFound exceptions.
But, there
>>>>>>> is no job execution taking place.
>>>>>>>
>>>>>>> After sometime, one by one, each goes down and the cluster shuts
>>>>>>> down.
>>>>>>>
>>>>>>> On Fri, Sep 19, 2014 at 2:15 PM, Tobias Pfeiffer <tgp@preferred.jp>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On Fri, Sep 19, 2014 at 5:17 PM, rapelly kartheek <
>>>>>>>> kartheek.mbms@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> > ,
>>>>>>>>>
>>>>>>>>> * you have copied a lot of files from various hosts to
>>>>>>>>> username@slave3:path*
>>>>>>>>> only from one node to all the other nodes...
>>>>>>>>>
>>>>>>>>
>>>>>>>> I don't think rsync can do that in one command as you described.
My
>>>>>>>> guess is that now you have a wild mixture of jar files all
across your
>>>>>>>> cluster which will lead to fancy exceptions like MethodNotFound
etc.,
>>>>>>>> that's maybe why your cluster is not working correctly.
>>>>>>>>
>>>>>>>> Tobias
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message