spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Haoyuan Li <haoyuan...@gmail.com>
Subject Re: deployment of spark on mesos and data locality in tachyon/hdfs
Date Wed, 01 Apr 2015 15:40:47 GMT
Response inline.

On Tue, Mar 31, 2015 at 10:41 PM, Sean Bigdatafun <sean.bigdatafun@gmail.com
> wrote:

> (resending...)
>
> I was thinking the same setup… But the more I think of this problem, and
> the more interesting this could be.
>
> If we allocate 50% total memory to Tachyon statically, then the Mesos
> benefits of dynamically scheduling resources go away altogether.
>

People can still benefits from Mesos' dynamically scheduling of the rest
memory as well as compute resource.


>
> Can Tachyon be resource managed by Mesos (dynamically)? Any thought or
> comment?
>


This requires some integration work.

Best,

Haoyuan


>
> Sean
>>
>>
>>
>>
>>
>> >Hi Haoyuan,
>>
>> >So on each mesos slave node I should allocate/section off some amount
>> >of memory for tachyon (let's say 50% of the total memory) and the rest
>> >for regular mesos tasks?
>>
>> >This means, on each slave node I would have tachyon worker (+ hdfs
>> >configuration to talk to s3 or the hdfs datanode) and the mesos slave
>> ?process. Is this correct?
>>
>>
>>
>
>
> --
> --Sean
>
>



-- 
Haoyuan Li
AMPLab, EECS, UC Berkeley
http://www.cs.berkeley.edu/~haoyuan/

Mime
View raw message