hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13113) Enable parallel test execution for hadoop-aws.
Date Tue, 10 May 2016 12:44:12 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15278034#comment-15278034

Steve Loughran commented on HADOOP-13113:

I think our tests generate race conditions more than most other bits of code do, as they are
repeatedly working on the same bits of the objects store and trying to do lots of small operations.

Looks like you are hitting the delayed-listing-consistency problem which S3 has —here on
a delete, rather than an addition.

why not have that test not fail immediately if the root dir isn't empty, but sleep and spin
for a bit until the dir empties up.

> Enable parallel test execution for hadoop-aws.
> ----------------------------------------------
>                 Key: HADOOP-13113
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13113
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HADOOP-13113.001.patch, HADOOP-13113.002.patch
> The full hadoop-aws test suite takes ~30 minutes to execute.  The tests spend most of
their time blocked on network I/O with the S3 back-end, but they don't saturate the bandwidth
of the NIC.  We can improve overall execution time by enabling parallel test execution.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message