spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <>
Subject Re: AMP Camp 3 cluster setup
Date Wed, 28 Aug 2013 23:28:55 GMT
By the way, an important note: Make sure you *shut down* your cluster after using it. Otherwise,
Amazon will keep charging you money for it! I've seen some people get caught by that in the

For others following this list, it's probably fine to start the cluster tomorrow morning (Pacific
time) instead of today.


On Aug 28, 2013, at 3:21 PM, seanm <> wrote:

> Shivaram-
> I just spun up a cluster and poked around a bit, it worked great!
> Thank you!
> Sean
> On Wednesday, August 28, 2013 3:08:04 PM UTC-7, Shivaram wrote:
> Hi 
> As a part of AMP Camp 3 to be held on Aug 29, 30th we will be hosting 
> hands-on exercises that will allow users to try out Spark, Shark, 
> Mesos, MLBase, BlinkDB etc. 
> Here are some notes on how to setup a cluster in case you want to 
> follow along with our live stream  NOTE: The exercises have not been 
> released yet and will be made available during the camp. You can 
> register to be notified at 
> To launch a cluster, the steps involved are: 
> 1. Clone the training-scripts git repo: 
> git clone 
> 2. Launch the cluster using spark-ec2 
> cd training-scripts 
> ./spark-ec2 -i <key_file> -k <name_of_key_pair> --copy launch amplab-training

> Where <name_of_key_pair> is the name of your EC2 key pair (that you 
> gave it when you created it), <key_file> is the private key file for 
> your key pair. 
> The scripts will launch 6 m1.xlarge machines (1 master + 5 slaves) and 
> each machine will have two EBS volumes attached to it. 
> This command may take a 30-40 minutes or longer and should produce a 
> bunch of output as it first spins up the nodes for your cluster, sets 
> up BDAS on them, and performs a large file copy of the datasets we’ll 
> use in the camp from an EBS volume to your instance of HDFS. 
> Let me know if you have any questions ! 
> Thanks 
> Shivaram 
> -- 
> You received this message because you are subscribed to the Google Groups "Spark Users"
> To unsubscribe from this group and stop receiving emails from it, send an email to
> For more options, visit

View raw message