spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Han JU <ju.han.fe...@gmail.com>
Subject Re: mounting SSD devices of EC2 r3.8xlarge instances
Date Wed, 04 Jun 2014 08:14:46 GMT
For SSDs in r3, maybe it's better to mount with `discard` option since it
supports TRIM:

What I did for r3.large:

  echo '/dev/xvdb /mnt ext4 defaults,noatime,nodiratime,discard 0 0' >>
/etc/fstab
  mkfs.ext4 /dev/xvdb
  mount /dev/xvdb



2014-06-03 19:15 GMT+02:00 Matei Zaharia <matei.zaharia@gmail.com>:

> Those instance types are not yet supported by the scripts, but
> https://issues.apache.org/jira/browse/SPARK-1790 is tracking this issue
> and it will soon be fixed in both branch-0.9 and 1.0. The problem is that
> those drives are not formatted on r3 machines, whereas they are on the
> other instance types, so the script assumed that they’d be formatted. You
> can manually go and do this to set them up:
>
> mkfs.ext4 /dev/sdb
> mkfs.ext4 /dev/sdc
> mount -o noatime /dev/sdb /mnt
> mount -o noatime /dev/sdc /mnt2
>
> Matei
>
> On Jun 3, 2014, at 10:05 AM, Andras Barjak <
> andras.barjak@lynxanalytics.com> wrote:
>
> > Hi,
> > I have noticed that upon launching a cluster consisting of r3.8xlarge
> high-memory instances the standard /mnt /mnt2 /mnt3 /mnt4 temporary
> directories get created and set up for temp usage, however they will point
> to the root 8Gb filesystem.
> > The 2x320GB SSD-s are not mounted and also they are not even formatted.
> >
> > This problem might affect other EC2 instances as well, I suppose.
> > I am using 0.9.1, is this something that has been corrected in the 1.0.0
> spark-ec2 script?
> >
> > regards,
> > András Barják
>
>


-- 
*JU Han*

Data Engineer @ Botify.com

+33 0619608888

Mime
View raw message