spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ankur Chauhan <achau...@brightcove.com>
Subject Re: deployment of spark on mesos and data locality in tachyon/hdfs
Date Tue, 31 Mar 2015 23:49:40 GMT
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Haoyuan,

So on each mesos slave node I should allocate/section off some amount
of memory for tachyon (let's say 50% of the total memory) and the rest
for regular mesos tasks?

This means, on each slave node I would have tachyon worker (+ hdfs
configuration to talk to s3 or the hdfs datanode) and the mesos slave
process. Is this correct?

On 31/03/2015 16:43, Haoyuan Li wrote:
> Tachyon should be co-located with Spark in this case.
> 
> Best,
> 
> Haoyuan
> 
> On Tue, Mar 31, 2015 at 4:30 PM, Ankur Chauhan
> <achauhan@brightcove.com <mailto:achauhan@brightcove.com>> wrote:
> 
> Hi,
> 
> I am fairly new to the spark ecosystem and I have been trying to
> setup a spark on mesos deployment. I can't seem to figure out the
> "best practices" around HDFS and Tachyon. The documentation about
> Spark's data-locality section seems to point that each of my mesos
> slave nodes should also run a hdfs datanode. This seems fine but I
> can't seem to figure out how I would go about pushing tachyon into
> the mix.
> 
> How should i organize my cluster? Should tachyon be colocated on my
> mesos worker nodes? or should all the spark jobs reach out to a
> separate hdfs/tachyon cluster.
> 
> -- Ankur Chauhan
> 
> ---------------------------------------------------------------------
>
> 
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> <mailto:user-unsubscribe@spark.apache.org> For additional commands,
> e-mail: user-help@spark.apache.org 
> <mailto:user-help@spark.apache.org>
> 
> 
> 
> 
> -- Haoyuan Li AMPLab, EECS, UC Berkeley 
> http://www.cs.berkeley.edu/~haoyuan/

- -- 
- -- Ankur Chauhan
-----BEGIN PGP SIGNATURE-----

iQEcBAEBAgAGBQJVGzKUAAoJEOSJAMhvLp3L3W4IAIVYiEKIZbC1a36/KWo94xYB
dvE4VXxF7z5FWmpuaHBEa+U1XWrR4cLVsQhocusOFn+oC7bstdltt3cGNAuwFSv6
Oogs4Sl1J4YZm8omKVdCkwD6Hv71HSntM8llz3qTW+Ljk2aKhfvNtp5nioQAm3e+
bs4ZKlCBij/xV3LbYYIePSS3lL0d9m1qEDJvi6jFcfm3gnBYeNeL9x92B5ylyth0
BGHnPN4sV/yopgrqOimLb12gSexHGNP1y6JBYy8NrHRY8SxkZ4sWKuyDnGDCOPOc
HC14Parf5Ly5FEz5g5WjF6HrXRdPlgr2ABxSLWOAB/siXsX9o/4yCy7NtDNcL6Y=
=f2xI
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message