hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andras Bokor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13082) RawLocalFileSystem does not fail when moving file to a non-existing directory
Date Thu, 05 May 2016 08:55:12 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272089#comment-15272089

Andras Bokor commented on HADOOP-13082:

[~stevel@apache.org], [~cnauroth],

Thank you guys for getting back to me.

First of all, yes, fallback logic was added way before HADOOP-9805.

I looked into it a bit deeper and it seems this rename situation is not that simple. We have
9 methods in FileSystemContractBaseTest which is broken and 6 of 9 is broken by fallback logic.
I have a small list about it (it was my notes during the debugging):
	Fallback creates the missing directory
	Move fails but fallback behavior throws exception. Test expects only a false
	Fallback creates the missing directory
	Fallback will throw an exception (cannot copy x to ist subdirectory y) but contract expects
just a false
	renameTo return false (which is good) but fallback throws exception while test does not expect
	rename returns with false but fallback throws exception

bq. I'm not sure now, after all, the raw OS doesn't do this, does it?
Yes, it does. In case of one volume mv command do just a rename. The inode will not change.
In case of multi-volume the OS does a copy-delete.

I checked Jakarta IO commons and Guava how do they handle this and they use the same solution
as we do in RawLocalFileSystem (if renameTo false, do a copy-delete). It is not good for us
because we do not know why the renameTo returned false (so we can't decide whether the false
was returned due to cross-volume move or other problem).
>From JDK 7 a new method was introduced. [Files.move|https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#move%28java.nio.file.Path,%20java.nio.file.Path,%20java.nio.file.CopyOption...%29]
do more or less the same as renameTo and it handles the cross-volume situation.

Now I am confused what to do:
# Skip this one and solve others
# Skip all the rename related contracts
# Solve all the broken contracts with FileUtil#copy
# Solve all the contracts with [Files.move|https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#move%28java.nio.file.Path,%20java.nio.file.Path,%20java.nio.file.CopyOption...%29]
# Other

What do you guys think?

> RawLocalFileSystem does not fail when moving file to a non-existing directory
> -----------------------------------------------------------------------------
>                 Key: HADOOP-13082
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13082
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Andras Bokor
>            Assignee: Andras Bokor
> FileSystemContractBaseTest#testRenameFileMoveToNonExistentDirectory: creates a file then
move it to a non-existing directory. It should fail but it will not (with RawLocalFileSystem)
because in RawLocalFileSystem#rename(Path, Path) method we have a fallback behavior that accomplishes
the rename by a full copy. The full copy will create the new directory and copy the file there.
> I see two possible solutions here:
> # Remove the fallback full copy behavior
> # Before full cp we should check whether the parent directory exists or not. If not return
false an do not do the full copy.
> The fallback logic was added by [HADOOP-9805|https://issues.apache.org/jira/browse/HADOOP-9805].

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message