spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: Using persistent hdfs on spark ec2 instanes
Date Thu, 23 Jan 2014 00:41:24 GMT
It should work correctly and yes, it starts and stops on port 9010.
You'll need to use "hdfs://<master-hostname>:9010/path/to/whatever" to
access files from Spark. Is that what you are asking about?

On Wed, Jan 22, 2014 at 4:36 PM, Aureliano Buendia <buendia360@gmail.com> wrote:
> peristent-hdfs server is set to 9010 port, instead of 9000. Does spark need
> more config for this?
>
>
> On Thu, Jan 23, 2014 at 12:26 AM, Patrick Wendell <pwendell@gmail.com>
> wrote:
>>
>> > 1. It seems by default spark ec2 uses ephemeral hdfs, how to switch this
>> > to
>> > persistent hdfs?
>> You can stop the ephemeral one using
>>
>> /root/ephemeral-hdfs/bin/stop-dfs.sh
>>
>> and start the persistent one using
>>
>>  /root/persistent-hdfs/bin/start-dfs.sh
>>
>> > 2. By default persistent hdfs server is not up, is this meant to be like
>> > this?
>>
>> Yes - it starts only an ephemeral one:
>>
>> "The spark-ec2 script already sets up a HDFS instance for you. It’s
>> installed in /root/ephemeral-hdfs"
>
>

Mime
View raw message