spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Challis <>
Subject Problem creating EC2 cluster using spark-ec2
Date Mon, 01 Dec 2014 15:34:50 GMT
I've been trying to create a Spark cluster on EC2 using the
documentation at
(with Spark 1.1.1).

Running the script successfully creates some EC2 instances, HDFS etc.,
but appears to fail to copy the actual files needed to run Spark

I ran the following commands:

$ cd ~/src/spark-1.1.1/ec2
$ ./spark-ec2 --key-pair=* --identity-file=* --slaves=1
--region=eu-west-1 --zone=eu-west-1a --instance-type=m3.medium
--no-ganglia launch foocluster

I see the following in the script's output:

(instance and HDFS set up happens here)
Persistent HDFS installed, won't start by default...
~/spark-ec2 ~/spark-ec2
Setting up spark-standalone
RSYNC'ing /root/spark/conf to slaves...
RSYNC'ing /root/spark-ec2 to slaves...
./spark-standalone/ line 22: /root/spark/sbin/ No
such file or directory
./spark-standalone/ line 27:
/root/spark/sbin/ No such file or directory
./spark-standalone/ line 33:
/root/spark/sbin/ No such file or directory
Setting up tachyon
RSYNC'ing /root/tachyon to slaves...
(Tachyon setup happens here without any problem)

I can ssh to the master (using the ./spark-ec2 login), and looking in
/root/, it contains:

$ ls /root
ephemeral-hdfs  hadoop-native  mapreduce  persistent-hdfs  scala
shark  spark  spark-ec2  tachyon

If I look in /root/spark (where the sbin directory should be found),
it only contains a single 'conf' directory:

$ ls /root/spark

Any idea why spark-ec2 might have failed to copy these files across?


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message