hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Radia (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11452) Revisit FileSystem.rename(path, path, options)
Date Fri, 06 Jan 2017 00:14:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15802980#comment-15802980

Sanjay Radia commented on HADOOP-11452:

*Some background:*

# FileSystem#rename(src, dest, options)  originally served as a temporary implementation of
FileContext#rename(src, dest, options). Hence it was kept protected. It was deprecated when
the implementation in AbstractFileSystem called dfsClient.rename(...) which btw is atomic
as noted in this Jira.
# Why didn't we make it public with the correct impl. The expectation was that all Hadoop
apps would move to FIleContext. The flaw in that plan was that apps like Hive neeed to run
on both Hadoop 1 and Hadoop 2 and they picked the lowest common denominator: Hadoop 1 which
only has FileSystem. We should have back ported  FileContext to Hadoop 1. Big mistake.

Going forward there is little chance of removing FileSystem since  many customers probably
use it directly. Hence I am okay with FileSystem#rename(src, dest, options) becoming public
and having a correct implementation (ie call  dfsClient#rename(...). However we should only
do this ONLY  if we feel customers/apps need the OVERWRITE flag.

I have already commented on the Rename.ATOMIC_REQUIRED. My vote is -1 for this option as I
explained in my comment above. We could move the  Rename.ATOMIC_REQUIRED part to a separate
jira  if folks want to discuss this further  and move forward on making rename-with-options

> Revisit FileSystem.rename(path, path, options)
> ----------------------------------------------
>                 Key: HADOOP-11452
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11452
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: fs
>    Affects Versions: 2.7.3
>            Reporter: Yi Liu
>            Assignee: Steve Loughran
>         Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected and with _deprecated_
annotation. And the default implementation is not atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a good and
atomic implementation. (Also an interesting thing in {{DFSClient}}, the _deprecated_ annotations
for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since it's atomic
for rename+overwrite, also it saves RPC calls if user desires rename+overwrite.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message