spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nishkam Ravi (JIRA)" <j...@apache.org>
Subject [jira] [Reopened] (SPARK-5423) ExternalAppendOnlyMap won't delete temp spilled file if some exception happens during using it
Date Wed, 04 Mar 2015 00:24:05 GMT

     [ https://issues.apache.org/jira/browse/SPARK-5423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Nishkam Ravi reopened SPARK-5423:
---------------------------------

The commit was reverted due to performance issues. Re-opening this JIRA.

> ExternalAppendOnlyMap won't delete temp spilled file if some exception happens during
using it 
> -----------------------------------------------------------------------------------------------
>
>                 Key: SPARK-5423
>                 URL: https://issues.apache.org/jira/browse/SPARK-5423
>             Project: Spark
>          Issue Type: Improvement
>          Components: Shuffle
>    Affects Versions: 1.0.0
>            Reporter: Shixiong Zhu
>            Assignee: Shixiong Zhu
>             Fix For: 1.3.0, 1.1.2, 1.2.2
>
>
> ExternalAppendOnlyMap won't delete temp spilled file if some exception happens during
using it.
> There is already a TODO in the comment:
> {code}
>     // TODO: Ensure this gets called even if the iterator isn't drained.
>     private def cleanup() {
>       batchIndex = batchOffsets.length  // Prevent reading any other batch
>       val ds = deserializeStream
>       deserializeStream = null
>       fileStream = null
>       ds.close()
>       file.delete()
>     }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message