spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nicholas Chammas (JIRA)" <>
Subject [jira] [Commented] (SPARK-5189) Reorganize EC2 scripts so that nodes can be provisioned independent of Spark master
Date Sat, 10 Jan 2015 19:43:34 GMT


Nicholas Chammas commented on SPARK-5189:

cc [~joshrosen] and [~shivaram] - What do y'all think?

> Reorganize EC2 scripts so that nodes can be provisioned independent of Spark master
> -----------------------------------------------------------------------------------
>                 Key: SPARK-5189
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: EC2
>            Reporter: Nicholas Chammas
> As of 1.2.0, we launch Spark clusters on EC2 by setting up the master first, then setting
up all the slaves together. This includes broadcasting files from the lonely master to potentially
hundreds of slaves.
> There are 2 main problems with this approach:
> # Broadcasting files from the master to all slaves using [{{copy-dir}}|]
(e.g. during [ephemeral-hdfs init|],
or during [Spark setup|])
takes a long time. This time increases as the number of slaves increases.
> # It's more complicated to add slaves to an existing cluster (a la [SPARK-2008]), since
slaves are only configured through the master during the setup of the master itself.
> Logically, the operations we want to implement are:
> * Provision a Spark node
> * Join a node to a cluster (including an empty cluster) as either a master or a slave
> * Remove a node from a cluster
> We need our scripts to roughly be organized to match the above operations. The goals
would be:
> # When launching a cluster, enable all cluster nodes to be provisioned in parallel, removing
the master-to-slave file broadcast bottleneck.
> # Facilitate cluster modifications like adding or removing nodes.
> # Enable exploration of infrastructure tools like [Terraform|]
that might simplify {{spark-ec2}} internals and perhaps even allow us to build [one tool that
launches Spark clusters on several different cloud platforms|!topic/terraform-tool/eD23GLLkfDw].
> More concretely, the modifications we need to make are:
> * Replace all occurrences of {{copy-dir}} or {{rsync}}-to-slaves with equivalent, slave-side
> * Repurpose {{}} as {{}} and make sure it fully
creates a node that can be used as either a master or slave.
> * Create a new script, {{}}, that takes a provisioned node, configures
it as a master or slave, and joins it to a cluster.
> * Move any remaining logic in {{}} up to {{}} and delete that script.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message