hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-13614) Purge some superfluous/obsolete S3 FS tests that are slowing test runs down
Date Wed, 19 Oct 2016 13:09:58 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Steve Loughran updated HADOOP-13614:
------------------------------------
    Attachment: HADOOP-13614-branch-2-007.patch

patch 007

* apply to branch-2
* verify that all appears well. 
* Tune more tests, including ITestS3AMiniYarnCluster.

A big change here is fixing up the scale tests to work as subclasses of the S3AScaleTestBase
indirect subclasses of AbstractFSContractTestBase, because that sets up the test timeout rule.
Rather than have a field of the same name and hope that its timeout gets picked up, I've tuned
how timeouts get set up, so the subclasses do it. All well and good, except those subclasses
are being called during the initialization of the base class, that is: before the subclasses
are full inited. I don't ever like doing that, though it is working here.

Tested S3a ireland, two (unrelated) failures
{code}
  ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive:97->Assert.fail:88
Deletion of child entries failed, still have1
  s3a://hwdev-steve-ireland-new/fork-5

  ITestS3AFileContextStatistics>FCStatisticsBaseTest.testStatistics:102->verifyWrittenBytes:54
expected:<512> but was:<0>
{code}

Time to run without scale tests, ~4 minutes, going to 6:30 with the scale tests @ default
size in.

> Purge some superfluous/obsolete S3 FS tests that are slowing test runs down
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-13614
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13614
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3, test
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>         Attachments: HADOOP-13614-branch-2-001.patch, HADOOP-13614-branch-2-002.patch,
HADOOP-13614-branch-2-002.patch, HADOOP-13614-branch-2-004.patch, HADOOP-13614-branch-2-005.patch,
HADOOP-13614-branch-2-006.patch, HADOOP-13614-branch-2-007.patch, testrun.txt
>
>
> Some of the slow test cases contain tests that are now obsoleted by newer ones. For example,
{{ITestS3ADeleteManyFiles}} has the test case {{testOpenCreate()}} which writes then reads
files up 25 MB.
> Have a look at which of the s3a tests are taking time, review them to see if newer tests
have superceded the slow ones; and cut them where appropriate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message