hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yi Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11452) Revisit FileSystem#rename
Date Sun, 04 Jan 2015 03:32:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14263741#comment-14263741

Yi Liu commented on HADOOP-11452:

I agree that {{rename}} should be defined to be atomic.
For HDFS, they are atomic. Yes, but for some other FS like S3, they are not atomic.
I also like to constrain the {{rename}} implementation to be atomic, and there are also other
applications assume the {{rename}} atomic if HDFS is expected configured as underlying fs,
such as in {{FileSystemRMStateStore}} of YARN. The trouble is some other FS like S3.

We have _no-overwrite-rename_ and _rename-with-options(overwrite)_, and the later is _protected_
and with _deprecated_ annotation today in {{FileSystem}} class. Should we make it public too?

> Revisit FileSystem#rename
> -------------------------
>                 Key: HADOOP-11452
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11452
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: fs
>            Reporter: Yi Liu
>            Assignee: Yi Liu
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected and with _deprecated_
annotation. And the default implementation is not atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a good and
atomic implementation. (Also an interesting thing in {{DFSClient}}, the _deprecated_ annotations
for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since it's atomic
for rename+overwrite, also it saves RPC calls if user desires rename+overwrite.

This message was sent by Atlassian JIRA

View raw message