spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Rosen (JIRA)" <>
Subject [jira] [Resolved] (SPARK-3105) Calling cache() after RDDs are pipelined has no effect in PySpark
Date Sat, 23 Jan 2016 23:23:39 GMT


Josh Rosen resolved SPARK-3105.
    Resolution: Won't Fix

Closing as "Won't Fix", since I don't think we have any plans to change this behavior in the
near term, nobody has complained about this, and it's a somewhat invasive tricky change.

> Calling cache() after RDDs are pipelined has no effect in PySpark
> -----------------------------------------------------------------
>                 Key: SPARK-3105
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 1.0.0, 1.1.0
>            Reporter: Josh Rosen
>            Assignee: Josh Rosen
> PySpark's PipelinedRDD decides whether to pipeline transformations by checking whether
those transformations are pipelinable _at the time that the PipelinedRDD objects are created_
rather than at the time that we invoke actions.  This might lead to problems if we call {{cache()}}
on an RDD after it's already been used in a pipeline:
> {code}
> rdd1 = sc.parallelize(range(100)).map(lambda x: x)
> rdd2 = x: 2 * x)
> rdd1.cache()
> rdd2.collect()
> {code}
> When I run this code, I'd expect {cache()}} to break the pipeline and cache intermediate
results, but instead the two transformations are pipelined together in Python, effectively
ignoring the {{cache()}}.
> Note that {{cache()}} works properly if we call it before performing any other transformations
on the RDD:
> {code}
> rdd1 = sc.parallelize(range(100)).map(lambda x: x).cache()
> rdd2 = x: 2 * x)
> rdd2.collect()
> {code}
> This works as expected and caches {{rdd1}}.
> To fix this, I think we dynamically decide whether to pipeline when we actually perform
actions, rather than statically deciding when we create the RDDs.
> We should also add tests for this.
> (Thanks to [~tdas] for pointing out this issue.)

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message