spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <...@preferred.jp>
Subject Re: rsync problem
Date Thu, 25 Sep 2014 01:22:39 GMT
Hi,

I assume you unintentionally did not reply to the list, so I'm adding it
back to CC.

How do you submit your job to the cluster?

Tobias


On Thu, Sep 25, 2014 at 2:21 AM, rapelly kartheek <kartheek.mbms@gmail.com>
wrote:

> How do I find out whether a node in the cluster is a master or slave??
> Till now I was thinking that slaves file under the conf folder makes the
> difference. Also, the MASTER_MASTER_IP in the spark-env.sh file.
>
> what else differentiates a slave from the master??
>
> On Wed, Sep 24, 2014 at 10:46 PM, rapelly kartheek <
> kartheek.mbms@gmail.com> wrote:
>
>> The job execution is taking place perfectly. Previously, all my print
>> statements used to be stored in spark/work/*/stdout file. But, now after
>> doing the rsync, I find that none of the prtint statements are getting
>> reflected in the stdout file under work folder. But, when I go to the code,
>> I find the statements in the code. But, they are not reflected into the
>> stdout file as before.
>>
>> Can you please tell me where I went wrong.  All I want is to see my
>> mofication in the code getting relected in output
>> .
>>
>> On Wed, Sep 24, 2014 at 10:22 PM, rapelly kartheek <
>> kartheek.mbms@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have a very important and fundamental doubt: I have rsynced the entire
>>> spark folder from the master to all slaves in the cluster. When I execute a
>>> job, its working perfectly. But, when I rsync the entire spark folder of
>>> the master to all the slaves, is it not that I am sending the  master
>>> configurations to all the slaves and making the slaves behave like master??
>>>
>>> First of all, is it correct to rsync the entire spark folder??
>>> But, if I change only one file, then how do I rsync it to all??
>>>
>>> On Fri, Sep 19, 2014 at 8:44 PM, rapelly kartheek <
>>> kartheek.mbms@gmail.com> wrote:
>>>
>>>> Thank you Soumya Simantha and Tobias. I've deleted the contents of the
>>>> work folder in all the nodes.
>>>> Now its working perfectly as it was before.
>>>>
>>>> Thank you
>>>> Karthik
>>>>
>>>> On Fri, Sep 19, 2014 at 4:46 PM, Soumya Simanta <
>>>> soumya.simanta@gmail.com> wrote:
>>>>
>>>>> One possible reason is maybe that the checkpointing directory
>>>>> $SPARK_HOME/work is rsynced as well.
>>>>> Try emptying the contents of the work folder on each node and try
>>>>> again.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Sep 19, 2014 at 4:53 AM, rapelly kartheek <
>>>>> kartheek.mbms@gmail.com> wrote:
>>>>>
>>>>>> I
>>>>>> * followed this command:rsync -avL --progress path/to/spark-1.0.0
>>>>>> username@destinationhostname:*
>>>>>>
>>>>>>
>>>>>> *path/to/destdirectory. Anyway, for now, I did it individually for
>>>>>> each node.*
>>>>>>
>>>>>> I have copied to each node at a time individually using the above
>>>>>> command. So, I guess the copying may not contain any mixture of files.
>>>>>> Also, as of now, I am not facing any MethodNotFound exceptions. But,
there
>>>>>> is no job execution taking place.
>>>>>>
>>>>>> After sometime, one by one, each goes down and the cluster shuts
>>>>>> down.
>>>>>>
>>>>>> On Fri, Sep 19, 2014 at 2:15 PM, Tobias Pfeiffer <tgp@preferred.jp>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> On Fri, Sep 19, 2014 at 5:17 PM, rapelly kartheek <
>>>>>>> kartheek.mbms@gmail.com> wrote:
>>>>>>>
>>>>>>>> > ,
>>>>>>>>
>>>>>>>> * you have copied a lot of files from various hosts to
>>>>>>>> username@slave3:path*
>>>>>>>> only from one node to all the other nodes...
>>>>>>>>
>>>>>>>
>>>>>>> I don't think rsync can do that in one command as you described.
My
>>>>>>> guess is that now you have a wild mixture of jar files all across
your
>>>>>>> cluster which will lead to fancy exceptions like MethodNotFound
etc.,
>>>>>>> that's maybe why your cluster is not working correctly.
>>>>>>>
>>>>>>> Tobias
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message