spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Challis <dave.chal...@aistemos.com>
Subject Problem creating EC2 cluster using spark-ec2
Date Mon, 01 Dec 2014 15:34:50 GMT
I've been trying to create a Spark cluster on EC2 using the
documentation at https://spark.apache.org/docs/latest/ec2-scripts.html
(with Spark 1.1.1).

Running the script successfully creates some EC2 instances, HDFS etc.,
but appears to fail to copy the actual files needed to run Spark
across.

I ran the following commands:

$ cd ~/src/spark-1.1.1/ec2
$ ./spark-ec2 --key-pair=* --identity-file=* --slaves=1
--region=eu-west-1 --zone=eu-west-1a --instance-type=m3.medium
--no-ganglia launch foocluster

I see the following in the script's output:

(instance and HDFS set up happens here)
...
Persistent HDFS installed, won't start by default...
~/spark-ec2 ~/spark-ec2
Setting up spark-standalone
RSYNC'ing /root/spark/conf to slaves...
*****.eu-west-1.compute.amazonaws.com
RSYNC'ing /root/spark-ec2 to slaves...
*****.eu-west-1.compute.amazonaws.com
./spark-standalone/setup.sh: line 22: /root/spark/sbin/stop-all.sh: No
such file or directory
./spark-standalone/setup.sh: line 27:
/root/spark/sbin/start-master.sh: No such file or directory
./spark-standalone/setup.sh: line 33:
/root/spark/sbin/start-slaves.sh: No such file or directory
Setting up tachyon
RSYNC'ing /root/tachyon to slaves...
...
(Tachyon setup happens here without any problem)

I can ssh to the master (using the ./spark-ec2 login), and looking in
/root/, it contains:

$ ls /root
ephemeral-hdfs  hadoop-native  mapreduce  persistent-hdfs  scala
shark  spark  spark-ec2  tachyon

If I look in /root/spark (where the sbin directory should be found),
it only contains a single 'conf' directory:

$ ls /root/spark
conf

Any idea why spark-ec2 might have failed to copy these files across?

Thanks,
Dave

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message