spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shao, Saisai" <saisai.s...@intel.com>
Subject RE: number of "Cached Partitions" v.s. "Total Partitions"
Date Tue, 22 Jul 2014 07:38:54 GMT
Yes, it's normal when memory is not enough to put the third partition, as you can see in your
attached picture.

Thanks
Jerry

From: Haopu Wang [mailto:HWang@qilinsoft.com]
Sent: Tuesday, July 22, 2014 3:09 PM
To: user@spark.apache.org
Subject: number of "Cached Partitions" v.s. "Total Partitions"


Hi, I'm using local mode and read a text file as RDD using JavaSparkContext.textFile() API.

And then call "cache()" method on the result RDD.



I look at the Storage information and find the RDD has 3 partitions but 2 of them have been
cached.

Is this a normal behavior? I assume all of partitions should be cached or none of them.

If I'm wrong, what are the cases when number of "cached" partitions is less than the total
number of partitions?



[cid:image001.jpg@01CFA5C3.0AE4B440]

Mime
View raw message