hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-15208) DistCp to offer option to save src/dest filesets as alternative to delete()
Date Mon, 05 Feb 2018 18:27:00 GMT
Steve Loughran created HADOOP-15208:
---------------------------------------

             Summary: DistCp to offer option to save src/dest filesets as alternative to delete()
                 Key: HADOOP-15208
                 URL: https://issues.apache.org/jira/browse/HADOOP-15208
             Project: Hadoop Common
          Issue Type: New Feature
          Components: tools/distcp
    Affects Versions: 2.9.0
            Reporter: Steve Loughran
            Assignee: Steve Loughran


There are opportunities to improve distcp delete performance and scalability with object stores,
but you need to test with production datasets to determine if the optimizations work, don't
run out of memory, etc.

By adding the option to save the sequence files of source, dest listings, people (myself included)
can experiment with different strategies before trying to commit one which doesn't scale



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message