hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ayush Saxena (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
Date Sat, 28 Mar 2020 08:36:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069312#comment-17069312

Ayush Saxena commented on HADOOP-16878:

Thanx [~gabor.bota] for the report and PR.
I tried this with a quick UT in HDFS. Seems I didn't get the exception thrown :
Sharing the UT :

  public void testCopyBetweenFsEqualPath() throws Exception {
    Configuration conf = getTestConfiguration();
    try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).build()) {
      final DistributedFileSystem dfs = cluster.getFileSystem();
      Path filePath = new Path("/dir/file");
      FileStatus fstatus = dfs.getFileStatus(filePath);
          () -> FileUtil.copy(dfs, fstatus, dfs, filePath, false, true, conf));
(Can place in this TestDistributedFileSystem.java)

The reason being :

   if (src.toUri().equals(dst.toUri())) {

In {{src}} the scheme is present but in {{dst}} it isn't present.

Can you give a check once.

> Copy command in FileUtil to throw an exception if the source and destination is the same
> ----------------------------------------------------------------------------------------
>                 Key: HADOOP-16878
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16878
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 3.3.0
>            Reporter: Gabor Bota
>            Assignee: Gabor Bota
>            Priority: Major
> We encountered an error during a test in our QE when the file destination and source
path were the same. This happened during an ADLS test, and there were no meaningful error
messages, so it was hard to find the root cause of the failure.
> The error we saw was that file size has changed during the copy operation. The new file
creation in the destination - which is the same as the source - creates a file and sets the
file length to zero. After this, getting the source file will fail because the sile size changed
during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if the source
and destination of the copy operation is the same, so debugging issues like this will be easier
in the future.

This message was sent by Atlassian Jira

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message