spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Siegmann <daniel.siegm...@velos.io>
Subject Re: Not fully cached when there is enough memory
Date Thu, 12 Jun 2014 20:14:15 GMT
I too have seen cached RDDs not hit 100%, even when they are DISK_ONLY.
Just saw that yesterday in fact. In some cases RDDs I expected didn't show
up in the list at all. I have no idea if this is an issue with Spark or
something I'm not understanding about how persist works (probably the
latter).

If I figure out the reason for this I'll let you know.


On Wed, Jun 11, 2014 at 8:54 PM, Shuo Xiang <shuoxiangpub@gmail.com> wrote:

> Xiangrui, clicking into the RDD link, it gives the same message, say only
> 96 of 100 partitions are cached. The disk/memory usage are the same, which
> is far below the limit.
> Is this what you want to check or other issue?
>
>
> On Wed, Jun 11, 2014 at 4:38 PM, Xiangrui Meng <mengxr@gmail.com> wrote:
>
>> Could you try to click one that RDD and see the storage info per
>> partition? I tried continuously caching RDDs, so new ones kick old
>> ones out when there is not enough memory. I saw similar glitches but
>> the storage info per partition is correct. If you find a way to
>> reproduce this error, please create a JIRA. Thanks! -Xiangrui
>>
>
>


-- 
Daniel Siegmann, Software Developer
Velos
Accelerating Machine Learning

440 NINTH AVENUE, 11TH FLOOR, NEW YORK, NY 10001
E: daniel.siegmann@velos.io W: www.velos.io

Mime
View raw message