hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bogdan Stolojan (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-17139) Re-enable optimized copyFromLocal implementation in S3AFileSystem
Date Wed, 02 Jun 2021 09:14:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17355593#comment-17355593
] 

Bogdan Stolojan commented on HADOOP-17139:
------------------------------------------

Would like to have a look at this one if that's ok

> Re-enable optimized copyFromLocal implementation in S3AFileSystem
> -----------------------------------------------------------------
>
>                 Key: HADOOP-17139
>                 URL: https://issues.apache.org/jira/browse/HADOOP-17139
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.3.0, 3.2.1
>            Reporter: Sahil Takiar
>            Priority: Minor
>
> It looks like HADOOP-15932 disabled the optimized copyFromLocal implementation in S3A
for correctness reasons.  innerCopyFromLocalFile should be fixed and re-enabled. The current
implementation uses FileSystem.copyFromLocal which will open an input stream from the local
fs and an output stream to the destination fs, and then call IOUtils.copyBytes. With default
configs, this will cause S3A to read the file into memory, write it back to a file on the
local fs, and then when the file is closed, upload it to S3.
> The optimized version of copyFromLocal in innerCopyFromLocalFile, directly creates a
PutObjectRequest request with the local file as the input.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message