hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15209) DistCp to eliminate needless deletion of files under already-deleted directories
Date Mon, 05 Mar 2018 20:29:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386708#comment-16386708
] 

Steve Loughran commented on HADOOP-15209:
-----------------------------------------

And manual distcp to s3 london of hadoop-auth followed by a mvn clean then a distcp -update
-delete
{code}
2018-03-05 19:29:31,084 [pool-6-thread-1] DEBUG s3a.S3AFileSystem (Listing.java:buildNextStatusBatch(486))
- Added 13 entries; ignored 0; hasNext=true; hasMoreObjects=false
2018-03-05 19:29:31,084 [pool-6-thread-1] DEBUG s3a.S3AFileSystem (Listing.java:sourceHasNext(378))
- Start iterating the provided status.
2018-03-05 19:29:31,088 [Thread-217] INFO  tools.SimpleCopyListing (SimpleCopyListing.java:printStats(608))
- Paths (files+dirs) cnt = 243; dirCnt = 58
2018-03-05 19:29:31,088 [Thread-217] INFO  tools.SimpleCopyListing (SimpleCopyListing.java:doBuildListing(402))
- Build file listing completed.
2018-03-05 19:29:31,109 [Thread-217] INFO  tools.DistCp (CopyListing.java:buildListing(94))
- Number of paths in the copy list: 243
2018-03-05 19:29:31,125 [Thread-217] INFO  tools.DistCp (CopyListing.java:buildListing(94))
- Number of paths in the copy list: 243
2018-03-05 19:29:31,142 [Thread-217] INFO  mapred.CopyCommitter (CopyCommitter.java:deleteMissing(414))
- Listing completed in 0:00:05.938
2018-03-05 19:29:31,147 [Thread-217] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- op_delete += 1  ->  1
2018-03-05 19:29:31,147 [Thread-217] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- op_get_file_status += 1  ->  481
2018-03-05 19:29:31,147 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(2106))
- Getting path status for s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes
 (distcp/hadoop-common-project/hadoop-auth/target/classes)
2018-03-05 19:29:31,147 [Thread-217] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  691
2018-03-05 19:29:31,165 [Thread-217] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  692
2018-03-05 19:29:31,198 [Thread-217] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_list_requests += 1  ->  260
2018-03-05 19:29:31,222 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2229))
- Found path as directory (with /): 1/0
2018-03-05 19:29:31,222 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2236))
- Prefix: distcp/hadoop-common-project/hadoop-auth/target/classes/META-INF/
2018-03-05 19:29:31,222 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1732))
- Delete path s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes
- recursive true
2018-03-05 19:29:31,222 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1737))
- delete: Path is a directory: s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes
2018-03-05 19:29:31,222 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1759))
- Getting objects for directory prefix distcp/hadoop-common-project/hadoop-auth/target/classes/
to delete
2018-03-05 19:29:31,222 [Thread-217] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_list_requests += 1  ->  261
2018-03-05 19:29:31,260 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/META-INF/LICENSE.txt
2018-03-05 19:29:31,260 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/META-INF/NOTICE.txt
2018-03-05 19:29:31,260 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticatedURL$1.class
2018-03-05 19:29:31,260 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticatedURL$AuthCookieHandler.class
2018-03-05 19:29:31,260 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticatedURL$Token.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticatedURL.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticationException.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/Authenticator.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/ConnectionConfigurator.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/KerberosAuthenticator$1.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/KerberosAuthenticator$KerberosConfiguration.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/PseudoAuthenticator.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AltKerberosAuthenticationHandler.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationFilter$1.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationFilter.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationHandler.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationHandlerUtil.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationToken.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/CompositeAuthenticationHandler.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/HttpConstants.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler$1.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler$2.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/LdapAuthenticationHandler$1.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/LdapAuthenticationHandler.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/MultiSchemeAuthenticationHandler.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/PseudoAuthenticationHandler.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/package-info.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/AuthToken.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/CertificateUtil.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosName$BadFormatString.class
2018-03-05 19:29:31,261 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosName$NoMatchingRule.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosName$Rule.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosName.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosUtil$DER.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosUtil.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/RandomSignerSecretProvider.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/RolloverSignerSecretProvider$1.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/RolloverSignerSecretProvider.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/Signer.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/SignerException.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/SignerSecretProvider.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider$1.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider$JaasConfiguration.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider$SASLOwnerACLProvider.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.class
2018-03-05 19:29:31,263 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1769))
- Got object to delete distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/util/PlatformName.class
2018-03-05 19:29:31,264 [Thread-217] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_delete_requests += 1  ->  1
2018-03-05 19:29:31,695 [Thread-217] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  693
2018-03-05 19:29:31,759 [Thread-217] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  694
2018-03-05 19:29:31,775 [Thread-217] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_list_requests += 1  ->  262
2018-03-05 19:29:31,799 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2229))
- Found path as directory (with /): 0/1
2018-03-05 19:29:31,799 [Thread-217] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2233))
- Summary: distcp/hadoop-common-project/hadoop-auth/target/.plxarc 30
2018-03-05 19:29:31,799 [Thread-217] INFO  mapred.CopyCommitter (CopyCommitter.java:deleteMissing(450))
- Deleted s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes
- missing at source
2018-03-05 19:29:31,800 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/META-INF
2018-03-05 19:29:31,800 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/META-INF/LICENSE.txt
2018-03-05 19:29:31,800 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/META-INF/NOTICE.txt
2018-03-05 19:29:31,800 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org
2018-03-05 19:29:31,800 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache
2018-03-05 19:29:31,800 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop
2018-03-05 19:29:31,800 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security
2018-03-05 19:29:31,800 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication
2018-03-05 19:29:31,800 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticatedURL$1.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticatedURL$AuthCookieHandler.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticatedURL$Token.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticatedURL.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/AuthenticationException.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/Authenticator.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/ConnectionConfigurator.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/KerberosAuthenticator$1.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/KerberosAuthenticator$KerberosConfiguration.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/client/PseudoAuthenticator.class
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server
2018-03-05 19:29:31,801 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AltKerberosAuthenticationHandler.class
2018-03-05 19:29:31,802 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationFilter$1.class
2018-03-05 19:29:31,802 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationFilter.class
2018-03-05 19:29:31,802 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationHandler.class
2018-03-05 19:29:31,802 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationHandlerUtil.class
2018-03-05 19:29:31,802 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/AuthenticationToken.class
2018-03-05 19:29:31,802 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/CompositeAuthenticationHandler.class
2018-03-05 19:29:31,802 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/HttpConstants.class
2018-03-05 19:29:31,802 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.class
2018-03-05 19:29:31,802 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler$1.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler$2.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/LdapAuthenticationHandler$1.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/LdapAuthenticationHandler.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/MultiSchemeAuthenticationHandler.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/PseudoAuthenticationHandler.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/server/package-info.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/AuthToken.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/CertificateUtil.class
2018-03-05 19:29:31,803 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosName$BadFormatString.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosName$NoMatchingRule.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosName$Rule.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosName.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosUtil$DER.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/KerberosUtil.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/RandomSignerSecretProvider.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/RolloverSignerSecretProvider$1.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/RolloverSignerSecretProvider.class
2018-03-05 19:29:31,804 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/Signer.class
2018-03-05 19:29:31,805 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/SignerException.class
2018-03-05 19:29:31,805 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/SignerSecretProvider.class
2018-03-05 19:29:31,805 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider$1.class
2018-03-05 19:29:31,805 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider$JaasConfiguration.class
2018-03-05 19:29:31,805 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider$SASLOwnerACLProvider.class
2018-03-05 19:29:31,805 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.class
2018-03-05 19:29:31,805 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/util
2018-03-05 19:29:31,805 [Thread-217] DEBUG mapred.CopyCommitter (CopyCommitter.java:deleteMissing(485))
- Skipping deletion of s3a://hwdev-steve-london/distcp/hadoop-common-project/hadoop-auth/target/classes/org/apache/hadoop/util/PlatformName.class
2018-03-05 19:29:31,808 [Thread-217] INFO  mapred.CopyCommitter (CopyCommitter.java:deleteMissing(497))
- Completed deletion of files from S3AFileSystem{uri=s3a://hwdev-steve-london, workingDir=s3a://hwdev-steve-london/user/stevel,
inputPolicy=normal, partSize=8388608, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=524288,
blockSize=33554432, multiPartThreshold=2147483647, serverSideEncryptionAlgorithm='NONE', blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@20796163,
metastore=NullMetadataStore, authoritative=false, useListV1=false, magicCommitter=true, boundedExecutor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=25,
available=25, waiting=0}, activeCount=0}, unboundedExecutor=java.util.concurrent.ThreadPoolExecutor@2e373567[Running,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], statistics {0 bytes
read, 0 bytes written, 544 read ops, 0 large read ops, 1 write ops}, metrics {{Context=s3aFileSystem}
{s3aFileSystemId=b0a84ec2-8ac0-4905-ade5-d608f1ea1e48} {bucket=hwdev-steve-london} {stream_opened=0}
{stream_close_operations=0} {stream_closed=0} {stream_aborted=0} {stream_seek_operations=0}
{stream_read_exceptions=0} {stream_forward_seek_operations=0} {stream_backward_seek_operations=0}
{stream_bytes_skipped_on_seek=0} {stream_bytes_backwards_on_seek=0} {stream_bytes_read=0}
{stream_read_operations=0} {stream_read_fully_operations=0} {stream_read_operations_incomplete=0}
{stream_bytes_read_in_close=0} {stream_bytes_discarded_in_abort=0} {files_created=0} {files_copied=0}
{files_copied_bytes=0} {files_deleted=50} {fake_directories_deleted=0} {directories_created=0}
{directories_deleted=0} {ignored_errors=0} {op_copy_from_local_file=0} {op_create=0} {op_create_non_recursive=0}
{op_delete=1} {op_exists=2} {op_get_file_status=481} {op_glob_status=3} {op_is_directory=0}
{op_is_file=0} {op_list_files=0} {op_list_located_status=0} {op_list_status=61} {op_mkdirs=47}
{op_open=0} {op_rename=0} {object_copy_requests=0} {object_delete_requests=1} {object_list_requests=262}
{object_continue_list_requests=0} {object_metadata_requests=694} {object_multipart_aborted=0}
{object_put_bytes=0} {object_put_requests=0} {object_put_requests_completed=0} {stream_write_failures=0}
{stream_write_block_uploads=0} {stream_write_block_uploads_committed=0} {stream_write_block_uploads_aborted=0}
{stream_write_total_time=0} {stream_write_total_data=0} {committer_commits_created=0} {committer_commits_completed=0}
{committer_jobs_completed=0} {committer_jobs_failed=0} {committer_tasks_completed=0} {committer_tasks_failed=0}
{committer_bytes_committed=0} {committer_bytes_uploaded=0} {committer_commits_failed=0} {committer_commits_aborted=0}
{committer_commits_reverted=0} {committer_magic_files_created=0} {s3guard_metadatastore_put_path_request=481}
{s3guard_metadatastore_initialization=0} {s3guard_metadatastore_retry=0} {s3guard_metadatastore_throttled=0}
{store_io_throttled=0} {object_put_requests_active=0} {object_put_bytes_pending=0} {stream_write_block_uploads_active=0}
{stream_write_block_uploads_pending=0} {stream_write_block_uploads_data_pending=0} {S3guard_metadatastore_put_path_latencyNumOps=12}
{S3guard_metadatastore_put_path_latency50thPercentileLatency=1621} {S3guard_metadatastore_put_path_latency75thPercentileLatency=1964}
{S3guard_metadatastore_put_path_latency90thPercentileLatency=2095} {S3guard_metadatastore_put_path_latency95thPercentileLatency=2287}
{S3guard_metadatastore_put_path_latency99thPercentileLatency=2287} {S3guard_metadatastore_throttle_rateNumEvents=0}
{S3guard_metadatastore_throttle_rate50thPercentileFrequency (Hz)=0} {S3guard_metadatastore_throttle_rate75thPercentileFrequency
(Hz)=0} {S3guard_metadatastore_throttle_rate90thPercentileFrequency (Hz)=0} {S3guard_metadatastore_throttle_rate95thPercentileFrequency
(Hz)=0} {S3guard_metadatastore_throttle_rate99thPercentileFrequency (Hz)=0} }}
2018-03-05 19:29:31,808 [Thread-217] INFO  mapred.CopyCommitter (CopyCommitter.java:deleteMissing(504))
- Deleted from target: files: 0 directories: 1; skipped 60
2018-03-05 19:29:31,808 [Thread-217] INFO  mapred.CopyCommitter (CopyCommitter.java:deleteMissing(507))
- Duration of deletions: 0:00:00.666
2018-03-05 19:29:31,808 [Thread-217] INFO  mapred.CopyCommitter (CopyCommitter.java:deleteMissing(509))
- Total duration of operation: 0:00:06.604
2018-03-05 19:29:31,809 [Thread-217] INFO  mapred.CopyCommitter (CopyCommitter.java:cleanup(185))
- Cleaning up temporary work folder: file:/tmp/hadoop/mapred/staging/stevel1049724566/.staging/_distcp2007172673
2018-03-05 19:29:32,289 [main] INFO  mapreduce.Job (Job.java:monitorAndPrintJob(1658)) - Job
job_local1086689566_0001 completed successfully
2018-03-05 19:29:32,297 [main] INFO  mapreduce.Job (Job.java:monitorAndPrintJob(1665)) - Counters:
24
	File System Counters
		FILE: Number of bytes read=1067879
		FILE: Number of bytes written=1847318
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		S3A: Number of bytes read=0
		S3A: Number of bytes written=0
		S3A: Number of read operations=731
		S3A: Number of large read operations=0
		S3A: Number of write operations=0
	Map-Reduce Framework
		Map input records=182
		Map output records=135
		Input split bytes=306
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=10
		Total committed heap usage (bytes)=839909376
	File Input Format Counters 
		Bytes Read=64532
	File Output Format Counters 
		Bytes Written=22806
	DistCp Counters
		Bandwidth in Btyes=0
		Bytes Skipped=1493640
		DIR_COPY=47
		Files Skipped=135
{code}

note the S3 stats declres 731 read ops.; a {{) find . -print | wc -l }} gives the files of
hadoop auth src & target as 244, so there is still ~3x reads for every source files, even
with delete of child entries being skipped (which happens at the end of the job), and src
checks bypassed. 


> DistCp to eliminate needless deletion of files under already-deleted directories
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-15209
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15209
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: tools/distcp
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>         Attachments: HADOOP-15209-001.patch, HADOOP-15209-002.patch, HADOOP-15209-003.patch,
HADOOP-15209-004.patch, HADOOP-15209-005.patch
>
>
> DistCP issues a delete(file) request even if is underneath an already deleted directory.
This generates needless load on filesystems/object stores, and, if the store throttles delete,
can dramatically slow down the delete operation.
> If the distcp delete operation can build a history of deleted directories, then it will
know when it does not need to issue those deletes.
> Care is needed here to make sure that whatever structure is created does not overload
the heap of the process.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message