hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@cloudera.com.INVALID>
Subject Re: Intervention. Stabilizing Yetus (Attn. Azure)
Date Thu, 12 Nov 2020 11:08:26 GMT
I'll look at these tomorrow -I'd rather fix whatever problem there is
rather than roll back a JAR change which is inevitably going to come back.

As extra motivation: no commits -> hadoop-azure until this is fixed.

On Mon, 9 Nov 2020 at 21:19, Ayush Saxena <ayushtkn@gmail.com> wrote:

> The failing Azure tests are being tracked at HADOOP-17325
>
> https://issues.apache.org/jira/browse/HADOOP-17325
>
> On Mon, 9 Nov 2020 at 23:02, Ahmed Hussein <a@ahussein.me> wrote:
>
> > I created new Jiras for HDFS failures. Please consider doing the same for
> > Yarn and Azure.
> > For convenience, the list of failures in the qbt report is as follows:
> >
> > Test Result
> > <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/
> > >
> > (50
> > failures / -7)
> >
> >    -
> >
> >
> org.apache.hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination.testGetCachedDatanodeReport
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.federation.router/TestRouterRpcMultiDestination/testGetCachedDatanodeReport/
> > >
> >    -
> >
> >
> org.apache.hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination.testNamenodeMetrics
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.federation.router/TestRouterRpcMultiDestination/testNamenodeMetrics/
> > >
> >    -
> >
> >
> org.apache.hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination.testErasureCoding
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.federation.router/TestRouterRpcMultiDestination/testErasureCoding/
> > >
> >    -
> >
> >
> org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testMissBlocksWhenReregister
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestBPOfferService/testMissBlocksWhenReregister/
> > >
> >    -
> >
> org.apache.hadoop.yarn.sls.TestReservationSystemInvariants.testSimulatorRunning[Testing
> >    with: SYNTH,
> >
> >
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler,
> >    (nodeFile null)]
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.sls/TestReservationSystemInvariants/testSimulatorRunning_Testing_with__SYNTH__org_apache_hadoop_yarn_server_resourcemanager_scheduler_fair_FairScheduler___nodeFile_null__/
> > >
> >    -
> > org.apache.hadoop.yarn.sls.appmaster.TestAMSimulator.testAMSimulator[1]
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.sls.appmaster/TestAMSimulator/testAMSimulator_1_/
> > >
> >    -
> >
> >
> org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer.testTokenThreadTimeout
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.server.resourcemanager.security/TestDelegationTokenRenewer/testTokenThreadTimeout/
> > >
> >    -
> >
> >
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithOpportunisticContainers
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.applications.distributedshell/TestDistributedShell/testDSShellWithOpportunisticContainers/
> > >
> >    -
> >
> >
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithEnforceExecutionType
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.applications.distributedshell/TestDistributedShell/testDSShellWithEnforceExecutionType/
> > >
> >    - org.apache.hadoop.fs.azure.TestBlobMetadata.testFolderMetadata
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testFolderMetadata/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestBlobMetadata.testFirstContainerVersionMetadata
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testFirstContainerVersionMetadata/
> > >
> >    - org.apache.hadoop.fs.azure.TestBlobMetadata.testPermissionMetadata
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testPermissionMetadata/
> > >
> >    -
> org.apache.hadoop.fs.azure.TestBlobMetadata.testOldPermissionMetadata
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testOldPermissionMetadata/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testNoTempBlobsVisible
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemConcurrency/testNoTempBlobsVisible/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testLinkBlobs
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemConcurrency/testLinkBlobs/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatusRootDir
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testListStatusRootDir/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryMoveToExistingDirectory
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testRenameDirectoryMoveToExistingDirectory/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatus
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testListStatus/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryAsExistingDirectory
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testRenameDirectoryAsExistingDirectory/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameToDirWithSamePrefixAllowed
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testRenameToDirWithSamePrefixAllowed/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testLSRootDir
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testLSRootDir/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testDeleteRecursively
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemContractMocked/testDeleteRecursively/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck.testWasbFsck
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemFileNameCheck/testWasbFsck/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testChineseCharactersFolderRename
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testChineseCharactersFolderRename/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListingWithZeroByteRenameMetadata
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRedoRenameFolderInFolderListingWithZeroByteRenameMetadata/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListing
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRedoRenameFolderInFolderListing/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testUriEncoding
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testUriEncoding/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testDeepFileCreation
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testDeepFileCreation/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testListDirectory
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testListDirectory/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderRenameInProgress
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRedoRenameFolderRenameInProgress/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameFolder
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRenameFolder/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameImplicitFolder
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRenameImplicitFolder/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolder
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRedoRenameFolder/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testStoreDeleteFolder
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testStoreDeleteFolder/
> > >
> >    -
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRename
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRename/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testListStatus
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testListStatus/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testRenameDirectoryAsEmptyDirectory
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testRenameDirectoryAsEmptyDirectory/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testListStatusFilterWithSomeMatches
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testListStatusFilterWithSomeMatches/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testRenameDirectoryAsNonExistentDirectory
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testRenameDirectoryAsNonExistentDirectory/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testGlobStatusSomeMatchesInDirectories
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testGlobStatusSomeMatchesInDirectories/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testGlobStatusWithMultipleWildCardMatches
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testGlobStatusWithMultipleWildCardMatches/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testDeleteRecursively
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testDeleteRecursively/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testImplicitFolderDeleted
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testImplicitFolderDeleted/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testFileAndImplicitFolderSameName
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testFileAndImplicitFolderSameName/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testSetOwnerOnImplicitFolder
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testSetOwnerOnImplicitFolder/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testFileInImplicitFolderDeleted
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testFileInImplicitFolderDeleted/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testImplicitFolderListed
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testImplicitFolderListed/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testCreatingDeepFileCreatesExplicitFolder
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testCreatingDeepFileCreatesExplicitFolder/
> > >
> >    -
> >
> >
> org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testSetPermissionOnImplicitFolder
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testSetPermissionOnImplicitFolder/
> > >
> >    - org.apache.hadoop.fs.azure.TestWasbFsck.testDelete
> >    <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestWasbFsck/testDelete/
> > >
> >
> >
> > On Mon, Nov 9, 2020 at 11:24 AM Ahmed Hussein <a@ahussein.me> wrote:
> >
> > > Hello folks,
> > >
> > > Over the last month, there has been concern about the stability of
> > Hadoop.
> > > Looking at the latest QBT report (Nov 8th, 2020 1:39 AM)
> > > <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/#showFailuresLink
> > >,
> > > there were 50 failing tests, 41 of which are in "hadoop-azure" module.
> > > Thanks to the effort of the community, the yetus qbt report looks
> better
> > > by miles. However, it will be highly appreciated if some developers
> > > volunteer some time to take a look at the hadoop-azure.
> > >
> > > If tests in fs.azure are irrelevant to active contributors, then please
> > > consider disabling those tests to save resources and avoid side effects
> > of
> > > those failures on the other modules (busy CPUs, memory release, ports
> > > listening..etc).
> > >
> > > Thank you.
> > >
> > > --
> > > Best Regards,
> > >
> > > *Ahmed Hussein, PhD*
> > >
> >
> >
> > --
> > Best Regards,
> >
> > *Ahmed Hussein, PhD*
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message