spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nicholas Chammas (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-6220) Allow extended EC2 options to be passed through spark-ec2
Date Mon, 09 Mar 2015 03:28:38 GMT

     [ https://issues.apache.org/jira/browse/SPARK-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Nicholas Chammas updated SPARK-6220:
------------------------------------
    Description: 
There are many EC2 options exposed by the boto library that spark-ec2 uses. 

Over time, many of these EC2 options have been bubbled up here and there to become spark-ec2
options.

Examples:
* spot prices
* placement groups
* VPC, subnet, and security group assignments

It's likely that more and more EC2 options will trickle up like this to become spark-ec2 options.

While major options are well suited to this type of promotion, we should probably allow users
to pass through EC2 options they want to use through spark-ec2 in some generic way.

Let's add two options:
* {{--ec2-instance-option}} -> [{{boto::run_instances}}|http://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.connection.EC2Connection.run_instances]
* {{--ec2-spot-instance-option}} -> [{{boto::request_spot_instances}}|http://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.connection.EC2Connection.request_spot_instances]

Each option can be specified multiple times and is simply passed directly to the underlying
boto call.

For example:
{code}
spark-ec2 \
    ...
    --ec2-instance-option "instance_initiated_shutdown_behavior=terminate" \
    --ec2-instance-option "ebs_optimized=True"
{code}

I'm not sure about the exact syntax of the extended options, but something like this will
do the trick as long as it can be made to pass the options correctly to boto in most cases.


I followed the example of {{ssh}}, which supports multiple extended options similarly.

{code}
ssh -o LogLevel=ERROR -o UserKnowHostsFile=/dev/null ...
{code}

  was:
There are many EC2 options exposed by the boto library that spark-ec2 uses. 

Over time, many of these EC2 options have been bubbled up here and there to become spark-ec2
options.

Examples:
* spot prices
* placement groups
* VPC, subnet, and security group assignments

It's likely that more and more EC2 options will trickle up like this to become spark-ec2 options.

While major options are well suited to this type of promotion, we should probably allow users
to pass through EC2 options they want to use through spark-ec2 in some generic way.

Let's add two options:
* {{--ec2-instance-option}} -> [{{boto::run_instances}}|http://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.connection.EC2Connection.run_instances]
* {{--ec2-spot-instance-option}} -> [{{boto::request_spot_instances}}|http://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.connection.EC2Connection.request_spot_instances]

Each option can be specified multiple times and is simply passed directly to the underlying
boto call.

For example:
{code}
spark-ec2 \
    ...
    --ec2-instance-option "instance_initiated_shutdown_behavior=terminate" \
    --ec2-instance-option "ebs_optimized=True"
{code}

I'm not sure about the exact syntax of the extended options, but something like this will
do the trick.


I followed the example of {{ssh}}, which supports multiple extended options similarly.

{code}
ssh -o LogLevel=ERROR -o UserKnowHostsFile=/dev/null ...
{code}


> Allow extended EC2 options to be passed through spark-ec2
> ---------------------------------------------------------
>
>                 Key: SPARK-6220
>                 URL: https://issues.apache.org/jira/browse/SPARK-6220
>             Project: Spark
>          Issue Type: Improvement
>          Components: EC2
>            Reporter: Nicholas Chammas
>            Priority: Minor
>
> There are many EC2 options exposed by the boto library that spark-ec2 uses. 
> Over time, many of these EC2 options have been bubbled up here and there to become spark-ec2
options.
> Examples:
> * spot prices
> * placement groups
> * VPC, subnet, and security group assignments
> It's likely that more and more EC2 options will trickle up like this to become spark-ec2
options.
> While major options are well suited to this type of promotion, we should probably allow
users to pass through EC2 options they want to use through spark-ec2 in some generic way.
> Let's add two options:
> * {{--ec2-instance-option}} -> [{{boto::run_instances}}|http://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.connection.EC2Connection.run_instances]
> * {{--ec2-spot-instance-option}} -> [{{boto::request_spot_instances}}|http://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.connection.EC2Connection.request_spot_instances]
> Each option can be specified multiple times and is simply passed directly to the underlying
boto call.
> For example:
> {code}
> spark-ec2 \
>     ...
>     --ec2-instance-option "instance_initiated_shutdown_behavior=terminate" \
>     --ec2-instance-option "ebs_optimized=True"
> {code}
> I'm not sure about the exact syntax of the extended options, but something like this
will do the trick as long as it can be made to pass the options correctly to boto in most
cases.
> I followed the example of {{ssh}}, which supports multiple extended options similarly.
> {code}
> ssh -o LogLevel=ERROR -o UserKnowHostsFile=/dev/null ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message