spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Chammas <nicholas.cham...@gmail.com>
Subject Re: Down-scaling Spark on EC2 cluster
Date Wed, 23 Jul 2014 16:13:41 GMT
There is a JIRA issue to track adding such functionality to spark-ec2:
SPARK-2008 <https://issues.apache.org/jira/browse/SPARK-2008> - Enhance
spark-ec2 to be able to add and remove slaves to an existing cluster
‚Äč


On Wed, Jul 23, 2014 at 10:12 AM, Akhil Das <akhil@sigmoidanalytics.com>
wrote:

> Hi
>
> Currently this is not supported out of the Box. But you can of course
> add/remove workers in a running cluster. Better option would be to use a
> Mesos cluster where adding/removing nodes are quiet simple. But again, i
> believe adding new worker in the middle of a task won't give you better
> performance.
>
> Thanks
> Best Regards
>
>
> On Wed, Jul 23, 2014 at 6:36 PM, Shubhabrata <mail2shubho@gmail.com>
> wrote:
>
>> Hello,
>>
>> We plan to use Spark on EC2 for our data science pipeline. We successfully
>> manage to set up cluster as-well-as launch and run applications on
>> remote-clusters. However, to enhance scalability we would like to
>> implement
>> auto-scaling in EC2 for Spark applications. However, I did not find any
>> proper reference about this. For example when we launch training programs
>> that use Matlab scripts on EC2 cluster we do auto scaling by SQS. Can
>> anyone
>> please suggest what are the options for Spark ? This is especially more
>> important when we would downscaling by removing a machine (how graceful
>> can
>> it be if it is in the middle of a task).
>>
>> Thanks in advance.
>>
>> Shubhabrata
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Down-scaling-Spark-on-EC2-cluster-tp10494.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>
>

Mime
View raw message