spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: creating new ami image for spark ec2 commands
Date Fri, 06 Jun 2014 08:40:20 GMT
you can comment out this function and Create a new one which will return
your ami-id and the rest of the script will run fine.

def get_spark_ami(opts):
  instance_types = {
    "m1.small":    "pvm",
    "m1.medium":   "pvm",
    "m1.large":    "pvm",
    "m1.xlarge":   "pvm",
    "t1.micro":    "pvm",
    "c1.medium":   "pvm",
    "c1.xlarge":   "pvm",
    "m2.xlarge":   "pvm",
    "m2.2xlarge":  "pvm",
    "m2.4xlarge":  "pvm",
    "cc1.4xlarge": "hvm",
    "cc2.8xlarge": "hvm",
    "cg1.4xlarge": "hvm",
    "hs1.8xlarge": "hvm",
    "hi1.4xlarge": "hvm",
    "m3.xlarge":   "hvm",
    "m3.2xlarge":  "hvm",
    "cr1.8xlarge": "hvm",
    "i2.xlarge":   "hvm",
    "i2.2xlarge":  "hvm",
    "i2.4xlarge":  "hvm",
    "i2.8xlarge":  "hvm",
    "c3.large":    "pvm",
    "c3.xlarge":   "pvm",
    "c3.2xlarge":  "pvm",
    "c3.4xlarge":  "pvm",
    "c3.8xlarge":  "pvm"
  }
  if opts.instance_type in instance_types:
    instance_type = instance_types[opts.instance_type]
  else:
    instance_type = "pvm"
    print >> stderr,\
        "Don't recognize %s, assuming type is pvm" % opts.instance_type

  ami_path = "%s/%s/%s" % (AMI_PREFIX, opts.region, instance_type)
  try:
    ami = urllib2.urlopen(ami_path).read().strip()
    print "Spark AMI: " + ami
  except:
    print >> stderr, "Could not resolve AMI at: " + ami_path
    sys.exit(1)

  return ami

Thanks
Best Regards


On Fri, Jun 6, 2014 at 2:14 AM, Matt Work Coarr <mattcoarr.work@gmail.com>
wrote:

> How would I go about creating a new AMI image that I can use with the
> spark ec2 commands? I can't seem to find any documentation.  I'm looking
> for a list of steps that I'd need to perform to make an Amazon Linux image
> ready to be used by the spark ec2 tools.
>
> I've been reading through the spark 1.0.0 documentation, looking at the
> script itself (spark_ec2.py), and looking at the github project
> mesos/spark-ec2.
>
> From what I can tell, the spark_ec2.py script looks up the id of the AMI
> based on the region and machine type (hvm or pvm) using static content
> derived from the github repo mesos/spark-ec2.
>
> The spark ec2 script loads the AMI id from this base url:
> https://raw.github.com/mesos/spark-ec2/v2/ami-list
> (Which presumably comes from https://github.com/mesos/spark-ec2 )
>
> For instance, I'm working with us-east-1 and pvm, I'd end up with AMI id:
> ami-5bb18832
>
> Is there a list of instructions for how this AMI was created?  Assuming
> I'm starting with my own Amazon Linux image, what would I need to do to
> make it usable where I could pass that AMI id to spark_ec2.py rather than
> using the default spark-provided AMI?
>
> Thanks,
> Matt
>

Mime
View raw message