spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shubhabrata <mail2shu...@gmail.com>
Subject Down-scaling Spark on EC2 cluster
Date Wed, 23 Jul 2014 13:06:15 GMT
Hello,

We plan to use Spark on EC2 for our data science pipeline. We successfully
manage to set up cluster as-well-as launch and run applications on
remote-clusters. However, to enhance scalability we would like to implement
auto-scaling in EC2 for Spark applications. However, I did not find any
proper reference about this. For example when we launch training programs
that use Matlab scripts on EC2 cluster we do auto scaling by SQS. Can anyone
please suggest what are the options for Spark ? This is especially more
important when we would downscaling by removing a machine (how graceful can
it be if it is in the middle of a task).

Thanks in advance.

Shubhabrata



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Down-scaling-Spark-on-EC2-cluster-tp10494.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message