spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-1201) Do not materialize partitions whenever possible in BlockManager
Date Sun, 30 Mar 2014 04:13:21 GMT

     [ https://issues.apache.org/jira/browse/SPARK-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Patrick Wendell updated SPARK-1201:
-----------------------------------

    Reporter: Patrick Wendell  (was: Patrick Cogan)

> Do not materialize partitions whenever possible in BlockManager
> ---------------------------------------------------------------
>
>                 Key: SPARK-1201
>                 URL: https://issues.apache.org/jira/browse/SPARK-1201
>             Project: Apache Spark
>          Issue Type: New Feature
>          Components: Block Manager, Spark Core
>            Reporter: Patrick Wendell
>            Assignee: Andrew Or
>             Fix For: 1.0.0
>
>
> This is a slightly more complex version of SPARK-942 where we try to avoid unrolling
iterators in other situations where it is possible. SPARK-942 focused on the case where the
DISK_ONLY storeage level was used. There are other cases though, such as if data is stored
serialized and in memory and but there is not enough memory left to store the RDD.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message