spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Work Coarr <mattcoarr.w...@gmail.com>
Subject spark ec2 commandline tool error "VPC security groups may not be used for a non-VPC launch"
Date Mon, 19 May 2014 17:58:43 GMT
Hi, I'm attempting to run "spark-ec2 launch" on AWS.  My AWS instances
would be in our EC2 VPC (which seems to be causing a problem).

The two security groups MyClusterName-master and MyClusterName-slaves have
already been setup with the same ports open as the security group that
spark-ec2 tries to create.  (My company has security rules where I don't
have permissions to create security groups, so they have to be created by
someone else ahead of time.)

I'm getting the error VPC security groups may not be used for a non-VPC
launch" when I try to run "spark-ec2 launch".

Is there something I need to do to make spark-ec2 launch the master and
slave instances within the VPC?

Here's the command-line and the error that I get...

command-line (I've changed the clustername to something generic):

$SPARK_HOME/ec2/spark-ec2 --key-pair=MyKeyPair
'--identity-file=~/.ssh/id_mysshkey' --slaves=2 --instance-type=m3.large
--region=us-east-1 --zone=us-east-1a --ami=myami --spark-version

=0.9.1 launch MyClusterName


error:

ERROR:boto:400 Bad Request

ERROR:boto:<?xml version="1.0" encoding="UTF-8"?>

<Response><Errors><Error><Code>InvalidParameterCombination</Code><Message>VPC
security groups may not be used for a non-VPC
launch</Message></Error></Errors><RequestID>8374cac5-5869-4f38-a141-2fdaf3b18326</Requ

estID></Response>

Setting up security groups...

Searching for existing cluster MyClusterName...

Launching instances...

Traceback (most recent call last):

  File "./spark_ec2.py", line 806, in <module>

    main()

  File "./spark_ec2.py", line 799, in main

    real_main()

  File "./spark_ec2.py", line 682, in real_main

    conn, opts, cluster_name)

  File "./spark_ec2.py", line 344, in launch_cluster

    block_device_map = block_map)

  File
"/opt/spark-0.9.1-bin-hadoop1/ec2/third_party/boto-2.4.1.zip/boto-2.4.1/boto/ec2/image.py",
line 255, in run

  File
"/opt/spark-0.9.1-bin-hadoop1/ec2/third_party/boto-2.4.1.zip/boto-2.4.1/boto/ec2/connection.py",
line 678, in run_instances

  File
"/opt/spark-0.9.1-bin-hadoop1/ec2/third_party/boto-2.4.1.zip/boto-2.4.1/boto/connection.py",
line 925, in get_object

boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request

<?xml version="1.0" encoding="UTF-8"?>

<Response><Errors><Error><Code>InvalidParameterCombination</Code><Message>VPC
security groups may not be used for a non-VPC
launch</Message></Error></Errors><RequestID>8374cac5-5869-4f38-a141-2fdaf3b18326</RequestID></Response>

Thanks for your help!!

Matt

Mime
View raw message