hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-13560) S3A to support huge file writes and operations -with tests
Date Thu, 01 Sep 2016 17:19:20 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Steve Loughran updated HADOOP-13560:
    Attachment: HADOOP-13560-branch-2-001.patch

Patch 001
# scale test which can create a configurable file; good to try with small (<1 part) uploads
as well as medium, and uploads so big they bring your heap down.
# New Scale Test (prefix STest) likes lots of upload bandwidth. It's running non-parallelized...I
wonder if it should be left out of integration entirely.
# they are also good for demonstrating the failure modes of S3A bits
# Note how test methods explicitly ask JUnit to run in a given order. Allows the tests to
isolate their operations yet still work in sequence.
# The upload one does a no-op if the destination file of that size exists. It was meant to
let you skip that bit if it was already there...I think I'll pull that feature as it only
gets in the way.
$ lots of extra monitoring of what is going on inside S3A, including a gauge of active PUT
request counts and bytes pending. 
# more troubleshooting docs.
# The fast output stream will retry on errors during request completion and abort. 

* HADOOP-13569 S3AFastOutputStream to take ProgressListener in file create(); used in test
* HADOOP-13566 NPE in S3AFastOutputStream.write
* HADOOP-13567 S3AFileSystem to override getStoragetStatistics() and so serve up its statistics

> S3A to support huge file writes and operations -with tests
> ----------------------------------------------------------
>                 Key: HADOOP-13560
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13560
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>         Attachments: HADOOP-13560-branch-2-001.patch
> An AWS SDK [issue|https://github.com/aws/aws-sdk-java/issues/367] highlights that metadata
isn't copied on large copies.
> 1. Add a test to do that large copy/rname and verify that the copy really works
> 1. Verify that metadata makes it over.
> Verifying large file rename is important on its own, as it is needed for very large commit
operations for committers using rename

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message