spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-2490) StackOverflowError when RDD dependencies are too long
Date Fri, 01 Aug 2014 19:13:38 GMT

     [ https://issues.apache.org/jira/browse/SPARK-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Matei Zaharia resolved SPARK-2490.
----------------------------------

       Resolution: Fixed
    Fix Version/s: 1.1.0

> StackOverflowError when RDD dependencies are too long
> -----------------------------------------------------
>
>                 Key: SPARK-2490
>                 URL: https://issues.apache.org/jira/browse/SPARK-2490
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.0.0
>            Reporter: Liang-Chi Hsieh
>             Fix For: 1.1.0
>
>
> When performing some transformations on RDDs after many iterations, the dependencies
of RDDs could be very long. It can easily cause StackOverflowError when recursively visiting
these dependencies in Spark core. For example:
> var rdd = sc.makeRDD(Array(1))
> for (i <- 1 to 1000) {
>   rdd = rdd.coalesce(1).cache()
>   rdd.collect()
> }



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message