spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Chammas <nicholas.cham...@gmail.com>
Subject EC2 clusters ready in launch time + 30 seconds
Date Thu, 10 Jul 2014 22:05:48 GMT
Hi devs!

Right now it takes a non-trivial amount of time to launch EC2 clusters.
Part of this time is spent starting the EC2 instances, which is out of our
control. Another part of this time is spent installing stuff on and
configuring the instances. This, we can control.

I’d like to explore approaches to upgrading spark-ec2 so that launching a
cluster of any size generally takes only 30 seconds on top of the time to
launch the base EC2 instances. Since Amazon can launch instances
concurrently, I believe this means we should be able to launch a fully
operational Spark cluster of any size in constant time. Is that correct?

Do we already have an idea of what it would take to get to that point?

Nick
​

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message