spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: Using persistent hdfs on spark ec2 instanes
Date Thu, 23 Jan 2014 00:26:50 GMT
> 1. It seems by default spark ec2 uses ephemeral hdfs, how to switch this to
> persistent hdfs?
You can stop the ephemeral one using

/root/ephemeral-hdfs/bin/stop-dfs.sh

and start the persistent one using

 /root/persistent-hdfs/bin/start-dfs.sh

> 2. By default persistent hdfs server is not up, is this meant to be like
> this?

Yes - it starts only an ephemeral one:

"The spark-ec2 script already sets up a HDFS instance for you. It’s
installed in /root/ephemeral-hdfs"

Mime
View raw message