spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-5499) iterative computing with 1000 iterations causes stage failure
Date Fri, 30 Jan 2015 13:23:34 GMT

    [ https://issues.apache.org/jira/browse/SPARK-5499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298612#comment-14298612
] 

Sean Owen commented on SPARK-5499:
----------------------------------

Ah, that may be right. persist() should also break the lineage, but here you'd still be computing
the whole lineage all at once from the start before anything can persist. Yes, how about checkpoint()?

> iterative computing with 1000 iterations causes stage failure
> -------------------------------------------------------------
>
>                 Key: SPARK-5499
>                 URL: https://issues.apache.org/jira/browse/SPARK-5499
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.2.0
>            Reporter: Tien-Dung LE
>
> I got an error "org.apache.spark.SparkException: Job aborted due to stage failure: Task
serialization failed: java.lang.StackOverflowError" when executing an action with 1000 transformations.
> Here is a code snippet to re-produce the error:
> {code}
>   import org.apache.spark.rdd.RDD
>   var pair: RDD[(Long,Long)] = sc.parallelize(Array((1L,2L)))
>     var newPair: RDD[(Long,Long)] = null
>     for (i <- 1 to 1000) {
>       newPair = pair.map(_.swap)
>       pair = newPair
>     }
>     println("Count = " + pair.count())
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message