Hey Garry,

Have you verified that your particular VPC and subnet are open to the world? In particular, have you verified the route table attached to your VPC / subnet contains an internet gateway open to the public?

I've run into this issue myself recently and that was the problem for me.


2015-08-24 5:58 GMT-07:00 Robin East <robin.east@xense.co.uk>:
spark-ec2 is the way to go however you may need to debug connectivity issues. For example do you know that the servers were correctly setup in AWS and can you access each node using ssh? If no then you need to work out why (it’s not a spark issue). If yes then you will need to work out why ssh via the spark-ec2 script is not working.

I’ve used spark-ec2 successfully many times but have never used the —vpc-id and —subnet-id options and that may be the source of your problems, especially since it appears to be a hostname resolution issue. If you could confirm the above questions then maybe someone on the list can help diagnose the specific problem.

Robin East
Spark GraphX in Action Michael Malak and Robin East
Manning Publications Co.

On 24 Aug 2015, at 13:45, Garry Chen <gc92@cornell.edu> wrote:

So what is the best way to deploy spark cluster in EC2 environment any suggestions?
From: Akhil Das [mailto:akhil@sigmoidanalytics.com] 
Sent: Friday, August 21, 2015 4:27 PM
To: Garry Chen <gc92@cornell.edu>
Cc: user@spark.apache.org
Subject: Re: Spark ec2 lunch problem

It may happen that the version of spark-ec2 script you are using is buggy or sometime AWS have problem provisioning machines.

On Aug 21, 2015 7:56 AM, "Garry Chen" <gc92@cornell.edu> wrote:
Hi All,
                I am trying to lunch a spark ec2 cluster by running  spark-ec2 --key-pair=key --identity-file=my.pem --vpc-id=myvpc --subnet-id=subnet-011 --spark-version=1.4.1 launch spark-cluster but getting following message endless.  Please help.
Warning: SSH connection error. (This could be temporary.)
SSH return code: 255
SSH output: ssh: Could not resolve hostname : Name or service not known