spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-19167) .inprogress file can't be renamed if driver exit when OOM occured in AsynchronousListenerBus thread
Date Wed, 11 Jan 2017 10:27:58 GMT

     [ https://issues.apache.org/jira/browse/SPARK-19167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Owen resolved SPARK-19167.
-------------------------------
    Resolution: Invalid

If you're asking a question -- use user@spark.apache.org. Do you have a suggested change?
I think that in the case of OOM, some bets may be off. Cleanup is best effort. 
You can try calling sc.stop() too because that's good practice, and because it should block
until the other stop() call finishes anyway.

> .inprogress file can't be renamed if driver exit when OOM occured in AsynchronousListenerBus
thread
> ---------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-19167
>                 URL: https://issues.apache.org/jira/browse/SPARK-19167
>             Project: Spark
>          Issue Type: Bug
>            Reporter: xwebber
>
> I was running a spark application and OOM occured in AsynchronousListenerBus thread.
This will cause Utils.tryOrStopSparkContext(sparkContext) to do sc.stop(). And then my user
code catch the exception that sc is stopping, and this will cause spark driver shut down.
but at this time the sc.stop in AsynchronousListenerBus thread hasn't call _cleaner.foreach(_.stop())
yet. This will cause the file left on disk forever. can it be solved?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message