spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shivaram Venkataraman <shiva...@eecs.berkeley.edu>
Subject Re: EC2 clusters ready in launch time + 30 seconds
Date Mon, 14 Jul 2014 00:53:05 GMT
It should be possible to improve cluster launch time if we are careful
about what commands we run during setup. One way to do this would be to
walk down the list of things we do for cluster initialization and see if
there is anything we can do make things faster. Unfortunately this might be
pretty time consuming, but I don't know of a better strategy. The place to
start would be the setup.sh file at
https://github.com/mesos/spark-ec2/blob/v3/setup.sh

Here are some things that take a lot of time and could be improved:
1. Creating swap partitions on all machines. We could check if there is a
way to get EC2 to always mount a swap partition
2. Copying / syncing things across slaves. The copy-dir script is called
too many times right now and each time it pauses for a few milliseconds
between slaves [1]. This could be improved by removing unnecessary copies
3. We could make less frequently used modules like Tachyon, persistent hdfs
not a part of the default setup.

[1] https://github.com/mesos/spark-ec2/blob/v3/copy-dir.sh#L42

Thanks
Shivaram




On Sat, Jul 12, 2014 at 7:02 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:

> On Thu, Jul 10, 2014 at 8:10 PM, Nate D'Amico <nate@reactor8.com> wrote:
>
> > Starting to work through some automation/config stuff for spark stack on
> > EC2 with a project, will be focusing the work through the apache bigtop
> > effort to start, can then share with spark community directly as things
> > progress if people are interested
>
>
> Let us know how that goes. I'm definitely interested in hearing more.
>
> Nick
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message