hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7973) DistributedFileSystem close has severe consequences
Date Fri, 13 Jan 2012 19:16:40 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185769#comment-13185769

Daryn Sharp commented on HADOOP-7973:

I don't think it's related to use_ip, but I'll double check.  Even if it is related, it's
still a bug.  I think using a finalizer avoids the problem you describe where something gets
a fs, passes it to others, then closes it.  Or will a finalizer introduce other problems?

bq.  App is expected to write the correct code - that is if the file system is closed, do
not use it post that. However I am not sure if apps do handle this correctly.

Let's say I'm using DFS.  I use some other package that opens the same default DFS, does something,
and then closes it.  Whatever I was doing before I called the external routine is now invalidated.
 What if I was writing to an output stream?  How would apps be able to reasonably recover
from their fs being unexpectedly closed when there's not really an error?  Or am I misunderstanding
your intent?

bq.  Other alternative solution is to disable cache

Isn't this going to cause multiple sockets to the same namenode?  For code that doesn't explicitly
call close (perhaps because of this bug), that results in leaked fds that tie up resources.
 I vaguely seem to recall that disabling the cache for a job would cause the JT(?) to overwhelm
the namenode with connections.

> DistributedFileSystem close has severe consequences
> ---------------------------------------------------
>                 Key: HADOOP-7973
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7973
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 1.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Blocker
>         Attachments: HADOOP-7973.patch
> The way {{FileSystem#close}} works is very problematic.  Since the {{FileSystems}} are
cached, any {{close}} by any caller will cause problems for every other reference to it. 
Will add more detail in the comments.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message