spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-8617) Handle history files better
Date Tue, 06 Dec 2016 21:23:58 GMT

     [ https://issues.apache.org/jira/browse/SPARK-8617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-8617:
-----------------------------------

    Assignee: Apache Spark

> Handle history files better
> ---------------------------
>
>                 Key: SPARK-8617
>                 URL: https://issues.apache.org/jira/browse/SPARK-8617
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 1.4.0
>            Reporter: Thomas Graves
>            Assignee: Apache Spark
>
> Right now there are many conditions in which history files get left around with the .inprogress
extension.  The cleaner doesn't remove these because it can't distinguish between something
running and left over abandoned files. 
> There are cases its really hard to not leave these around (application killed/crashes/etc).
But we should still try to handle the history files better. 
>   
> For instance we could first write them to the .sparkStaging directory and then moving
to the history file location before the application finishes.   That way you only get completed
history files.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message