flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-4910) Introduce safety net for closing file system streams
Date Thu, 27 Oct 2016 13:29:58 GMT

    [ https://issues.apache.org/jira/browse/FLINK-4910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611879#comment-15611879
] 

ASF GitHub Bot commented on FLINK-4910:
---------------------------------------

Github user StephanEwen commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2691#discussion_r85334057
  
    --- Diff: flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java ---
    @@ -55,6 +60,40 @@
     
     	private static final String HADOOP_WRAPPER_SCHEME = "hdwrapper";
     
    +	private static final Logger LOG = LoggerFactory.getLogger(FileSystem.class);
    +
    +	/**
    +	 * Create a SafetyNetCloseableRegistry for a Task. This method should be called at the
beginning of the task's
    +	 * main thread.
    +	 */
    +	public static void createFileSystemCloseableRegistryForTask() {
    +		SafetyNetCloseableRegistry current = REGISTRIES.get();
    +		if (null != current) {
    +			if (!current.isClosed()) {
    +				IOUtils.closeQuietly(current);
    +				LOG.warn("Found open SafetyNetCloseableRegistry. Closed and replaced it.");
    +			}
    +		}
    +		REGISTRIES.set(new SafetyNetCloseableRegistry());
    +	}
    +
    +	/**
    +	 * Create a SafetyNetCloseableRegistry for a Task. This method should be called at the
end of the task's
    +	 * main thread or when the task should be canceled.
    +	 */
    +	public static void disposeFileSystemCloseableRegistryForTask() {
    +		SafetyNetCloseableRegistry ret = REGISTRIES.get();
    +		if (null != ret) {
    +			LOG.info("Closing SafetyNetCloseableRegistry.");
    --- End diff --
    
    Let's log something like "Ensuring all FileSystem streams are closed". Users will understand
that better ;-)


> Introduce safety net for closing file system streams
> ----------------------------------------------------
>
>                 Key: FLINK-4910
>                 URL: https://issues.apache.org/jira/browse/FLINK-4910
>             Project: Flink
>          Issue Type: Improvement
>            Reporter: Stefan Richter
>            Assignee: Stefan Richter
>
> Streams that are opened through {{FileSystem}} must be closed at the end of their life
cycle. However, we found hints that some code forgets to close such streams.
> We should introduce i) a mechanism that closes leaking unclosed streams after usage and
ii) provides logging that helps us to track down and fi the sources of such leaks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message