spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Chammas <nicholas.cham...@gmail.com>
Subject Re: Problem creating EC2 cluster using spark-ec2
Date Wed, 03 Dec 2014 01:11:06 GMT
Interesting. Do you have any problems when launching in us-east-1? What is
the full output of spark-ec2 when launching a cluster? (Post it to a gist
if it’s too big for email.)
​

On Mon, Dec 1, 2014 at 10:34 AM, Dave Challis <dave.challis@aistemos.com>
wrote:

> I've been trying to create a Spark cluster on EC2 using the
> documentation at https://spark.apache.org/docs/latest/ec2-scripts.html
> (with Spark 1.1.1).
>
> Running the script successfully creates some EC2 instances, HDFS etc.,
> but appears to fail to copy the actual files needed to run Spark
> across.
>
> I ran the following commands:
>
> $ cd ~/src/spark-1.1.1/ec2
> $ ./spark-ec2 --key-pair=* --identity-file=* --slaves=1
> --region=eu-west-1 --zone=eu-west-1a --instance-type=m3.medium
> --no-ganglia launch foocluster
>
> I see the following in the script's output:
>
> (instance and HDFS set up happens here)
> ...
> Persistent HDFS installed, won't start by default...
> ~/spark-ec2 ~/spark-ec2
> Setting up spark-standalone
> RSYNC'ing /root/spark/conf to slaves...
> *****.eu-west-1.compute.amazonaws.com
> RSYNC'ing /root/spark-ec2 to slaves...
> *****.eu-west-1.compute.amazonaws.com
> ./spark-standalone/setup.sh: line 22: /root/spark/sbin/stop-all.sh: No
> such file or directory
> ./spark-standalone/setup.sh: line 27:
> /root/spark/sbin/start-master.sh: No such file or directory
> ./spark-standalone/setup.sh: line 33:
> /root/spark/sbin/start-slaves.sh: No such file or directory
> Setting up tachyon
> RSYNC'ing /root/tachyon to slaves...
> ...
> (Tachyon setup happens here without any problem)
>
> I can ssh to the master (using the ./spark-ec2 login), and looking in
> /root/, it contains:
>
> $ ls /root
> ephemeral-hdfs  hadoop-native  mapreduce  persistent-hdfs  scala
> shark  spark  spark-ec2  tachyon
>
> If I look in /root/spark (where the sbin directory should be found),
> it only contains a single 'conf' directory:
>
> $ ls /root/spark
> conf
>
> Any idea why spark-ec2 might have failed to copy these files across?
>
> Thanks,
> Dave
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message