hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15281) Distcp to add no-rename copy option
Date Fri, 02 Mar 2018 18:17:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383909#comment-16383909
] 

Steve Loughran commented on HADOOP-15281:
-----------------------------------------

debug level log of a distcp of one file to s3a. 20 metadata requests get logged per file.
{code}
16:51:20,555 INFO  mapred.CopyMapper (CopyMapper.java:map(154)) - Copying file:/home/s/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
to s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
16:51:20,555 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- op_get_file_status += 1  ->  8987
16:51:20,555 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(2098)) - Getting
path status for s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
 (distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java)
16:51:20,555 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19643
16:51:20,591 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19644
16:51:20,626 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_list_requests += 1  ->  7689
16:51:20,664 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2245)) - Not Found:
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
16:51:20,664 INFO  mapred.RetriableFileCopyCommand (RetriableFileCopyCommand.java:getTmpFile(245))
- Creating temp file: s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
16:51:20,664 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- op_create += 1  ->  851
16:51:20,664 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- op_get_file_status += 1  ->  8988
16:51:20,664 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(2098)) - Getting
path status for s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
 (distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0)
16:51:20,664 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19645
16:51:20,697 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19646
16:51:20,732 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_list_requests += 1  ->  7690
16:51:20,770 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2245)) - Not Found:
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
16:51:20,770 DEBUG s3a.S3ABlockOutputStream (S3ABlockOutputStream.java:<init>(169))
- Initialized S3ABlockOutputStream for distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
output to FileBlock{index=1, destFile=/tmp/hadoop-stevel/s3a/s3ablock-0001-1407016756609004936.tmp,
state=Writing, dataSize=0, limit=8388608}
16:51:20,772 DEBUG s3a.S3ABlockOutputStream (S3ABlockOutputStream.java:close(349)) - S3ABlockOutputStream{WriteOperationHelper
{bucket=hwdev-steve-ireland-new}, blockSize=8388608, activeBlock=FileBlock{index=1, destFile=/tmp/hadoop-stevel/s3a/s3ablock-0001-1407016756609004936.tmp,
state=Writing, dataSize=16963, limit=8388608}}: Closing block #1: current block= FileBlock{index=1,
destFile=/tmp/hadoop-stevel/s3a/s3ablock-0001-1407016756609004936.tmp, state=Writing, dataSize=16963,
limit=8388608}
16:51:20,772 DEBUG s3a.S3ABlockOutputStream (S3ABlockOutputStream.java:putObject(413)) - Executing
regular upload for WriteOperationHelper {bucket=hwdev-steve-ireland-new}
16:51:20,772 DEBUG s3a.S3ADataBlocks (S3ADataBlocks.java:startUpload(324)) - Start datablock[1]
upload
16:51:20,772 DEBUG s3a.S3ADataBlocks (S3ADataBlocks.java:enterState(231)) - FileBlock{index=1,
destFile=/tmp/hadoop-stevel/s3a/s3ablock-0001-1407016756609004936.tmp, state=Writing, dataSize=16963,
limit=8388608}: entering state Upload
16:51:20,772 DEBUG s3a.S3ABlockOutputStream (S3ABlockOutputStream.java:clearActiveBlock(216))
- Clearing active block
16:51:20,772 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:putObjectDirect(1520))
- PUT 16963 bytes to distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
16:51:20,772 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:incrementPutStartStatistics(1583))
- PUT start 16963 bytes
16:51:20,772 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_put_requests += 1  ->  1146
16:51:20,969 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:incrementPutCompletedStatistics(1600))
- PUT completed success=true; 16963 bytes
16:51:20,969 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_put_bytes += 16963  ->  42252442
16:51:20,969 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_put_requests_completed += 1  ->  1146
16:51:20,969 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:finishedWrite(2662))
- Finished write to distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0,
len 16963
16:51:20,969 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_delete_requests += 1  ->  2846
16:51:21,017 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3ABlockOutputStream (S3AUtils.java:closeAll(1297))
- Closing org.apache.hadoop.fs.s3a.S3ADataBlocks$BlockUploadData@5dc0fde5
16:51:21,018 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3ABlockOutputStream (S3AUtils.java:closeAll(1297))
- Closing FileBlock{index=1, destFile=/tmp/hadoop-stevel/s3a/s3ablock-0001-1407016756609004936.tmp,
state=Upload, dataSize=16963, limit=8388608}
16:51:21,018 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3ADataBlocks (S3ADataBlocks.java:enterState(231))
- FileBlock{index=1, destFile=/tmp/hadoop-stevel/s3a/s3ablock-0001-1407016756609004936.tmp,
state=Upload, dataSize=16963, limit=8388608}: entering state Closed
16:51:21,018 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3ADataBlocks (S3ADataBlocks.java:close(346))
- Closed FileBlock{index=1, destFile=/tmp/hadoop-stevel/s3a/s3ablock-0001-1407016756609004936.tmp,
state=Closed, dataSize=16963, limit=8388608}
16:51:21,018 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3ADataBlocks (S3ADataBlocks.java:innerClose(886))
- Closing FileBlock{index=1, destFile=/tmp/hadoop-stevel/s3a/s3ablock-0001-1407016756609004936.tmp,
state=Closed, dataSize=16963, limit=8388608}
16:51:21,018 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3ADataBlocks (S3ADataBlocks.java:closeBlock(940))
- block[1]: closeBlock()
16:51:21,018 DEBUG s3a.S3ABlockOutputStream (S3ABlockOutputStream.java:close(389)) - Upload
complete to distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
by WriteOperationHelper {bucket=hwdev-steve-ireland-new}
16:51:21,019 DEBUG s3a.S3ABlockOutputStream (S3AUtils.java:closeAll(1297)) - Closing FileBlock{index=1,
destFile=/tmp/hadoop-stevel/s3a/s3ablock-0001-1407016756609004936.tmp, state=Closed, dataSize=16963,
limit=8388608}
16:51:21,019 DEBUG s3a.S3ABlockOutputStream (S3AUtils.java:closeAll(1297)) - Closing org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@5bfba522
16:51:21,019 DEBUG s3a.S3ABlockOutputStream (S3ABlockOutputStream.java:close(395)) - Statistics:
OutputStreamStatistics{blocksSubmitted=1, blocksInQueue=1, blocksActive=0, blockUploadsCompleted=0,
blockUploadsFailed=0, bytesPendingUpload=0, bytesUploaded=16963, blocksAllocated=1, blocksReleased=1,
blocksActivelyAllocated=0, exceptionsInMultipartFinalize=0, transferDuration=0 ms, queueDuration=0
ms, averageQueueTime=0 ms, totalUploadDuration=0 ms, effectiveBandwidth=0.0 bytes/s}
16:51:21,019 DEBUG s3a.S3ABlockOutputStream (S3AUtils.java:closeAll(1297)) - Closing OutputStreamStatistics{blocksSubmitted=1,
blocksInQueue=1, blocksActive=0, blockUploadsCompleted=0, blockUploadsFailed=0, bytesPendingUpload=0,
bytesUploaded=16963, blocksAllocated=1, blocksReleased=1, blocksActivelyAllocated=0, exceptionsInMultipartFinalize=0,
transferDuration=0 ms, queueDuration=0 ms, averageQueueTime=0 ms, totalUploadDuration=0 ms,
effectiveBandwidth=0.0 bytes/s}
16:51:21,019 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileChecksum(2990)) - getFileChecksum(s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0)
16:51:21,019 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19647
16:51:21,193 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- op_exists += 1  ->  1702
16:51:21,193 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- op_get_file_status += 1  ->  8989
16:51:21,193 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(2098)) - Getting
path status for s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
 (distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java)
16:51:21,193 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19648
16:51:21,230 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19649
16:51:21,264 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_list_requests += 1  ->  7691
16:51:21,303 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2245)) - Not Found:
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
16:51:21,303 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- op_exists += 1  ->  1703
16:51:21,303 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- op_get_file_status += 1  ->  8990
16:51:21,303 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(2098)) - Getting
path status for s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom
 (distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom)
16:51:21,303 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19650
16:51:21,340 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19651
16:51:21,397 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2188)) - Found file
(with /): fake directory
16:51:21,397 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerRename(889)) - Rename path s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
to s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
16:51:21,397 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- op_rename += 1  ->  851
16:51:21,397 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- op_get_file_status += 1  ->  8991
16:51:21,397 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(2098)) - Getting
path status for s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
 (distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0)
16:51:21,397 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19652
16:51:21,433 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2166)) - Found exact
file: normal file
16:51:21,433 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- op_get_file_status += 1  ->  8992
16:51:21,433 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(2098)) - Getting
path status for s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
 (distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java)
16:51:21,433 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19653
16:51:21,466 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19654
16:51:21,500 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_list_requests += 1  ->  7692
16:51:21,537 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2245)) - Not Found:
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
16:51:21,537 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerRename(943)) - rename: destination
path s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
not found
16:51:21,537 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- op_get_file_status += 1  ->  8993
16:51:21,537 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(2098)) - Getting
path status for s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom
 (distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom)
16:51:21,537 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19655
16:51:21,571 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19656
16:51:21,606 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2188)) - Found file
(with /): fake directory
16:51:21,606 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerRename(973)) - rename: renaming
file s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
to s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
16:51:21,606 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:copyFile(2507)) - copyFile distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
-> distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java

16:51:21,606 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19657
16:51:21,851 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1724)) - Delete path
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
- recursive false
16:51:21,852 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1781)) - delete: Path
is a file
16:51:21,852 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_delete_requests += 1  ->  2847
16:51:21,890 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerRename(1065)) - source &
dest parents are different; fix up dir markers
16:51:21,890 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_delete_requests += 1  ->  2848
16:51:21,990 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19658
16:51:22,143 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19659
16:51:22,180 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_list_requests += 1  ->  7693
16:51:22,223 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2221)) - Found path
as directory (with /): 0/1
16:51:22,223 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2225)) - Summary:
distcp/hadoop-common-project/.DS_Store 6148
16:51:22,223 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- op_delete += 1  ->  851
16:51:22,224 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- op_get_file_status += 1  ->  8994
16:51:22,224 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(2098)) - Getting
path status for s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
 (distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0)
16:51:22,224 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19660
16:51:22,260 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19661
16:51:22,295 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_list_requests += 1  ->  7694
16:51:22,335 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2245)) - Not Found:
s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
16:51:22,335 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:delete(1700)) - Couldn't delete s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/.distcp.tmp.attempt_local9668494_0001_m_000000_0
- does not exist
16:51:22,335 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- op_get_file_status += 1  ->  8995
16:51:22,335 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(2098)) - Getting
path status for s3a://hwdev-steve-ireland-new/distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java
 (distcp/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/bloom/BloomFilterCommonTester.java)
16:51:22,335 DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- object_metadata_requests += 1  ->  19662
16:51:22,378 DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2166)) - Found exact
file: normal file
{code}

> Distcp to add no-rename copy option
> -----------------------------------
>
>                 Key: HADOOP-15281
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15281
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: tools/distcp
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>
> Currently Distcp uploads a file by two strategies
> # append parts
> # copy to temp then rename
> option 2 executes the following swquence in {{promoteTmpToTarget}}
> {code}
>     if ((fs.exists(target) && !fs.delete(target, false))
>         || (!fs.exists(target.getParent()) && !fs.mkdirs(target.getParent()))
>         || !fs.rename(tmpTarget, target)) {
>       throw new IOException("Failed to promote tmp-file:" + tmpTarget
>                               + " to: " + target);
>     }
> {code}
> For any object store, that's a lot of HTTP requests; for S3A you are looking at 12+ requests
and an O(data) copy call. 
> This is not a good upload strategy for any store which manifests its output atomically
at the end of the write().
> Proposed: add a switch to write direct to the dest path. either a conf option or a CLI
option



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message