hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ayush Saxena <ayush...@gmail.com>
Subject Re: Intervention. Stabilizing Yetus (Attn. Azure)
Date Mon, 09 Nov 2020 21:18:55 GMT
The failing Azure tests are being tracked at HADOOP-17325

https://issues.apache.org/jira/browse/HADOOP-17325

On Mon, 9 Nov 2020 at 23:02, Ahmed Hussein <a@ahussein.me> wrote:

> I created new Jiras for HDFS failures. Please consider doing the same for
> Yarn and Azure.
> For convenience, the list of failures in the qbt report is as follows:
>
> Test Result
> <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/
> >
> (50
> failures / -7)
>
>    -
>
>  org.apache.hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination.testGetCachedDatanodeReport
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.federation.router/TestRouterRpcMultiDestination/testGetCachedDatanodeReport/
> >
>    -
>
>  org.apache.hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination.testNamenodeMetrics
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.federation.router/TestRouterRpcMultiDestination/testNamenodeMetrics/
> >
>    -
>
>  org.apache.hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination.testErasureCoding
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.federation.router/TestRouterRpcMultiDestination/testErasureCoding/
> >
>    -
>
>  org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testMissBlocksWhenReregister
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestBPOfferService/testMissBlocksWhenReregister/
> >
>    -
> org.apache.hadoop.yarn.sls.TestReservationSystemInvariants.testSimulatorRunning[Testing
>    with: SYNTH,
>
>  org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler,
>    (nodeFile null)]
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.sls/TestReservationSystemInvariants/testSimulatorRunning_Testing_with__SYNTH__org_apache_hadoop_yarn_server_resourcemanager_scheduler_fair_FairScheduler___nodeFile_null__/
> >
>    -
> org.apache.hadoop.yarn.sls.appmaster.TestAMSimulator.testAMSimulator[1]
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.sls.appmaster/TestAMSimulator/testAMSimulator_1_/
> >
>    -
>
>  org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer.testTokenThreadTimeout
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.server.resourcemanager.security/TestDelegationTokenRenewer/testTokenThreadTimeout/
> >
>    -
>
>  org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithOpportunisticContainers
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.applications.distributedshell/TestDistributedShell/testDSShellWithOpportunisticContainers/
> >
>    -
>
>  org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithEnforceExecutionType
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.applications.distributedshell/TestDistributedShell/testDSShellWithEnforceExecutionType/
> >
>    - org.apache.hadoop.fs.azure.TestBlobMetadata.testFolderMetadata
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testFolderMetadata/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestBlobMetadata.testFirstContainerVersionMetadata
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testFirstContainerVersionMetadata/
> >
>    - org.apache.hadoop.fs.azure.TestBlobMetadata.testPermissionMetadata
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testPermissionMetadata/
> >
>    - org.apache.hadoop.fs.azure.TestBlobMetadata.testOldPermissionMetadata
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testOldPermissionMetadata/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testNoTempBlobsVisible
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemConcurrency/testNoTempBlobsVisible/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testLinkBlobs
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemConcurrency/testLinkBlobs/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatusRootDir
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testListStatusRootDir/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryMoveToExistingDirectory
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testRenameDirectoryMoveToExistingDirectory/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatus
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testListStatus/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryAsExistingDirectory
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testRenameDirectoryAsExistingDirectory/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameToDirWithSamePrefixAllowed
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testRenameToDirWithSamePrefixAllowed/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testLSRootDir
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testLSRootDir/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testDeleteRecursively
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testDeleteRecursively/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck.testWasbFsck
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemFileNameCheck/testWasbFsck/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testChineseCharactersFolderRename
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testChineseCharactersFolderRename/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListingWithZeroByteRenameMetadata
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRedoRenameFolderInFolderListingWithZeroByteRenameMetadata/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListing
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRedoRenameFolderInFolderListing/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testUriEncoding
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testUriEncoding/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testDeepFileCreation
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testDeepFileCreation/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testListDirectory
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testListDirectory/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderRenameInProgress
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRedoRenameFolderRenameInProgress/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameFolder
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRenameFolder/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameImplicitFolder
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRenameImplicitFolder/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolder
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRedoRenameFolder/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testStoreDeleteFolder
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testStoreDeleteFolder/
> >
>    - org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRename
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRename/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testListStatus
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testListStatus/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testRenameDirectoryAsEmptyDirectory
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testRenameDirectoryAsEmptyDirectory/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testListStatusFilterWithSomeMatches
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testListStatusFilterWithSomeMatches/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testRenameDirectoryAsNonExistentDirectory
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testRenameDirectoryAsNonExistentDirectory/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testGlobStatusSomeMatchesInDirectories
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testGlobStatusSomeMatchesInDirectories/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testGlobStatusWithMultipleWildCardMatches
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testGlobStatusWithMultipleWildCardMatches/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testDeleteRecursively
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testDeleteRecursively/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testImplicitFolderDeleted
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testImplicitFolderDeleted/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testFileAndImplicitFolderSameName
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testFileAndImplicitFolderSameName/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testSetOwnerOnImplicitFolder
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testSetOwnerOnImplicitFolder/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testFileInImplicitFolderDeleted
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testFileInImplicitFolderDeleted/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testImplicitFolderListed
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testImplicitFolderListed/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testCreatingDeepFileCreatesExplicitFolder
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testCreatingDeepFileCreatesExplicitFolder/
> >
>    -
>
>  org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testSetPermissionOnImplicitFolder
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testSetPermissionOnImplicitFolder/
> >
>    - org.apache.hadoop.fs.azure.TestWasbFsck.testDelete
>    <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestWasbFsck/testDelete/
> >
>
>
> On Mon, Nov 9, 2020 at 11:24 AM Ahmed Hussein <a@ahussein.me> wrote:
>
> > Hello folks,
> >
> > Over the last month, there has been concern about the stability of
> Hadoop.
> > Looking at the latest QBT report (Nov 8th, 2020 1:39 AM)
> > <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/#showFailuresLink
> >,
> > there were 50 failing tests, 41 of which are in "hadoop-azure" module.
> > Thanks to the effort of the community, the yetus qbt report looks better
> > by miles. However, it will be highly appreciated if some developers
> > volunteer some time to take a look at the hadoop-azure.
> >
> > If tests in fs.azure are irrelevant to active contributors, then please
> > consider disabling those tests to save resources and avoid side effects
> of
> > those failures on the other modules (busy CPUs, memory release, ports
> > listening..etc).
> >
> > Thank you.
> >
> > --
> > Best Regards,
> >
> > *Ahmed Hussein, PhD*
> >
>
>
> --
> Best Regards,
>
> *Ahmed Hussein, PhD*
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message