hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ahmed Hussein...@ahussein.me>
Subject Re: Intervention. Stabilizing Yetus (Attn. Azure)
Date Mon, 09 Nov 2020 17:31:40 GMT
I created new Jiras for HDFS failures. Please consider doing the same for
Yarn and Azure.
For convenience, the list of failures in the qbt report is as follows:

Test Result
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/>
(50
failures / -7)

   -
   org.apache.hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination.testGetCachedDatanodeReport
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.federation.router/TestRouterRpcMultiDestination/testGetCachedDatanodeReport/>
   -
   org.apache.hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination.testNamenodeMetrics
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.federation.router/TestRouterRpcMultiDestination/testNamenodeMetrics/>
   -
   org.apache.hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination.testErasureCoding
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.federation.router/TestRouterRpcMultiDestination/testErasureCoding/>
   -
   org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testMissBlocksWhenReregister
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestBPOfferService/testMissBlocksWhenReregister/>
   - org.apache.hadoop.yarn.sls.TestReservationSystemInvariants.testSimulatorRunning[Testing
   with: SYNTH,
   org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler,
   (nodeFile null)]
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.sls/TestReservationSystemInvariants/testSimulatorRunning_Testing_with__SYNTH__org_apache_hadoop_yarn_server_resourcemanager_scheduler_fair_FairScheduler___nodeFile_null__/>
   - org.apache.hadoop.yarn.sls.appmaster.TestAMSimulator.testAMSimulator[1]
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.sls.appmaster/TestAMSimulator/testAMSimulator_1_/>
   -
   org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer.testTokenThreadTimeout
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.server.resourcemanager.security/TestDelegationTokenRenewer/testTokenThreadTimeout/>
   -
   org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithOpportunisticContainers
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.applications.distributedshell/TestDistributedShell/testDSShellWithOpportunisticContainers/>
   -
   org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithEnforceExecutionType
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.applications.distributedshell/TestDistributedShell/testDSShellWithEnforceExecutionType/>
   - org.apache.hadoop.fs.azure.TestBlobMetadata.testFolderMetadata
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testFolderMetadata/>
   -
   org.apache.hadoop.fs.azure.TestBlobMetadata.testFirstContainerVersionMetadata
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testFirstContainerVersionMetadata/>
   - org.apache.hadoop.fs.azure.TestBlobMetadata.testPermissionMetadata
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testPermissionMetadata/>
   - org.apache.hadoop.fs.azure.TestBlobMetadata.testOldPermissionMetadata
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testOldPermissionMetadata/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testNoTempBlobsVisible
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemConcurrency/testNoTempBlobsVisible/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testLinkBlobs
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemConcurrency/testLinkBlobs/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatusRootDir
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testListStatusRootDir/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryMoveToExistingDirectory
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testRenameDirectoryMoveToExistingDirectory/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatus
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testListStatus/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryAsExistingDirectory
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testRenameDirectoryAsExistingDirectory/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameToDirWithSamePrefixAllowed
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testRenameToDirWithSamePrefixAllowed/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testLSRootDir
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testLSRootDir/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testDeleteRecursively
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testDeleteRecursively/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck.testWasbFsck
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemFileNameCheck/testWasbFsck/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testChineseCharactersFolderRename
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testChineseCharactersFolderRename/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListingWithZeroByteRenameMetadata
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRedoRenameFolderInFolderListingWithZeroByteRenameMetadata/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListing
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRedoRenameFolderInFolderListing/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testUriEncoding
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testUriEncoding/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testDeepFileCreation
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testDeepFileCreation/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testListDirectory
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testListDirectory/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderRenameInProgress
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRedoRenameFolderRenameInProgress/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameFolder
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRenameFolder/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameImplicitFolder
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRenameImplicitFolder/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolder
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRedoRenameFolder/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testStoreDeleteFolder
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testStoreDeleteFolder/>
   - org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRename
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRename/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testListStatus
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testListStatus/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testRenameDirectoryAsEmptyDirectory
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testRenameDirectoryAsEmptyDirectory/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testListStatusFilterWithSomeMatches
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testListStatusFilterWithSomeMatches/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testRenameDirectoryAsNonExistentDirectory
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testRenameDirectoryAsNonExistentDirectory/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testGlobStatusSomeMatchesInDirectories
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testGlobStatusSomeMatchesInDirectories/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testGlobStatusWithMultipleWildCardMatches
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testGlobStatusWithMultipleWildCardMatches/>
   -
   org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testDeleteRecursively
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testDeleteRecursively/>
   -
   org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testImplicitFolderDeleted
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testImplicitFolderDeleted/>
   -
   org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testFileAndImplicitFolderSameName
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testFileAndImplicitFolderSameName/>
   -
   org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testSetOwnerOnImplicitFolder
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testSetOwnerOnImplicitFolder/>
   -
   org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testFileInImplicitFolderDeleted
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testFileInImplicitFolderDeleted/>
   -
   org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testImplicitFolderListed
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testImplicitFolderListed/>
   -
   org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testCreatingDeepFileCreatesExplicitFolder
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testCreatingDeepFileCreatesExplicitFolder/>
   -
   org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testSetPermissionOnImplicitFolder
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testSetPermissionOnImplicitFolder/>
   - org.apache.hadoop.fs.azure.TestWasbFsck.testDelete
   <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestWasbFsck/testDelete/>


On Mon, Nov 9, 2020 at 11:24 AM Ahmed Hussein <a@ahussein.me> wrote:

> Hello folks,
>
> Over the last month, there has been concern about the stability of Hadoop.
> Looking at the latest QBT report (Nov 8th, 2020 1:39 AM)
> <https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/#showFailuresLink>,
> there were 50 failing tests, 41 of which are in "hadoop-azure" module.
> Thanks to the effort of the community, the yetus qbt report looks better
> by miles. However, it will be highly appreciated if some developers
> volunteer some time to take a look at the hadoop-azure.
>
> If tests in fs.azure are irrelevant to active contributors, then please
> consider disabling those tests to save resources and avoid side effects of
> those failures on the other modules (busy CPUs, memory release, ports
> listening..etc).
>
> Thank you.
>
> --
> Best Regards,
>
> *Ahmed Hussein, PhD*
>


-- 
Best Regards,

*Ahmed Hussein, PhD*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message