hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer ...@apache.org>
Subject Fwd: Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64
Date Thu, 15 Mar 2018 17:15:41 GMT

	For my part of the HDFS bug bash, I’ve gotten the ASF Windows build working again. Starting
tomorrow, results will be sent to the *-dev lists.

A few notes:

* It only runs the unit tests.  There’s not much point in running the other Yetus plugins
since those are covered by the Linux one and this build is slow enough as it is.

* There are two types of ASF build nodes: Windows Server 2012 and Windows Server 2016. This
job can run on both and will use whichever one has a free slot.

* It ALWAYS applies HADOOP-14667.05.patch prior to running.  As a result, this is only set
up for trunk with no parameterization to run other branches.

* The URI handling for file paths in hadoop-common and elsewhere is pretty broken on Windows,
so many many many unit tests are failing and I wouldn't be surprised if Windows hadoop installs
are horked as a result.

* Runtime is about 12-13 hours with many tests taking significantly longer than their UNIX
counterparts.  My guess is that this caused by winutils.  Changing from winutils to Java 7
API calls would get this more in line and be a significant performance boost for Windows clients/servers
as well.

Have fun.

=====

For more details, see https://builds.apache.org/job/hadoop-trunk-win/406/ <https://builds.apache.org/job/hadoop-trunk-win/406/>

[Mar 14, 2018 6:26:58 PM] (xyao) HDFS-13251. Avoid using hard coded datanode data dirs in
unit tests.
[Mar 14, 2018 8:05:24 PM] (jlowe) MAPREDUCE-7064. Flaky test
[Mar 14, 2018 8:14:36 PM] (inigoiri) HDFS-13198. RBF: RouterHeartbeatService throws out CachedStateStore
[Mar 14, 2018 8:36:53 PM] (wangda) Revert "HADOOP-13707. If kerberos is enabled while HTTP
SPNEGO is not
[Mar 14, 2018 10:47:56 PM] (fabbri) HADOOP-15278 log s3a at info. Contributed by Steve Loughran.




-1 overall


The following subsystems voted -1:
   unit


The following subsystems are considered long running:
(runtime bigger than 1h 00m 00s)
   unit


Specific tests:

   Failed CTEST tests :

      test_test_libhdfs_threaded_hdfs_static 

   Failed junit tests :

      hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec 
      hadoop.fs.contract.rawlocal.TestRawlocalContractAppend 
      hadoop.fs.TestFsShellCopy 
      hadoop.fs.TestFsShellList 
      hadoop.fs.TestLocalFileSystem 
      hadoop.http.TestHttpServer 
      hadoop.http.TestHttpServerLogs 
      hadoop.io.compress.TestCodec 
      hadoop.io.nativeio.TestNativeIO 
      hadoop.ipc.TestSocketFactory 
      hadoop.metrics2.impl.TestStatsDMetrics 
      hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal 
      hadoop.security.TestSecurityUtil 
      hadoop.security.TestShellBasedUnixGroupsMapping 
      hadoop.security.token.TestDtUtilShell 
      hadoop.util.TestNativeCodeLoader 
      hadoop.fs.TestWebHdfsFileContextMainOperations 
      hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy 
      hadoop.hdfs.crypto.TestHdfsCryptoStreams 
      hadoop.hdfs.qjournal.client.TestQuorumJournalManager 
      hadoop.hdfs.qjournal.server.TestJournalNode 
      hadoop.hdfs.qjournal.server.TestJournalNodeSync 
      hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks 
      hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped 
      hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages 
      hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks 
      hadoop.hdfs.server.blockmanagement.TestReplicationPolicy 
      hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl 
      hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles 
      hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory 
      hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy 
      hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement 
      hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery 
      hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter 
      hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl 
      hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation 
      hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica 
      hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage 
      hadoop.hdfs.server.datanode.TestBlockRecovery 
      hadoop.hdfs.server.datanode.TestBlockScanner 
      hadoop.hdfs.server.datanode.TestDataNodeFaultInjector 
      hadoop.hdfs.server.datanode.TestDataNodeMetrics 
      hadoop.hdfs.server.datanode.TestDataNodeUUID 
      hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
      hadoop.hdfs.server.datanode.TestDirectoryScanner 
      hadoop.hdfs.server.datanode.TestHSync 
      hadoop.hdfs.server.datanode.web.TestDatanodeHttpXFrame 
      hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand 
      hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC 
      hadoop.hdfs.server.federation.router.TestRouterAdminCLI 
      hadoop.hdfs.server.mover.TestStorageMover 
      hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA 
      hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA 
      hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics 
      hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot 
      hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot 
      hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots 
      hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff 
      hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport 
      hadoop.hdfs.server.namenode.TestAddBlock 
      hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands 
      hadoop.hdfs.server.namenode.TestCheckpoint 
      hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate 
      hadoop.hdfs.server.namenode.TestEditLogRace 
      hadoop.hdfs.server.namenode.TestFileTruncate 
      hadoop.hdfs.server.namenode.TestFsck 
      hadoop.hdfs.server.namenode.TestFSImage 
      hadoop.hdfs.server.namenode.TestFSImageWithSnapshot 
      hadoop.hdfs.server.namenode.TestNamenodeCapacityReport 
      hadoop.hdfs.server.namenode.TestNameNodeMXBean 
      hadoop.hdfs.server.namenode.TestNestedEncryptionZones 
      hadoop.hdfs.server.namenode.TestQuotaByStorageType 
      hadoop.hdfs.server.namenode.TestStartup 
      hadoop.hdfs.TestDatanodeRegistration 
      hadoop.hdfs.TestDecommission 
      hadoop.hdfs.TestDFSOutputStream 
      hadoop.hdfs.TestDFSShell 
      hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy 
      hadoop.hdfs.TestDFSStripedOutputStreamWithFailure 
      hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy 
      hadoop.hdfs.TestFetchImage 
      hadoop.hdfs.TestFileAppend 
      hadoop.hdfs.TestFileConcurrentReader 
      hadoop.hdfs.TestFileCreation 
      hadoop.hdfs.TestFileLengthOnClusterRestart 
      hadoop.hdfs.TestHDFSFileSystemContract 
      hadoop.hdfs.TestHDFSServerPorts 
      hadoop.hdfs.TestHDFSTrash 
      hadoop.hdfs.TestHFlush 
      hadoop.hdfs.TestLeaseRecovery 
      hadoop.hdfs.TestLocalDFS 
      hadoop.hdfs.TestMaintenanceState 
      hadoop.hdfs.TestMiniDFSCluster 
      hadoop.hdfs.TestPread 
      hadoop.hdfs.TestQuota 
      hadoop.hdfs.TestReadStripedFileWithDecodingCorruptData 
      hadoop.hdfs.TestReadStripedFileWithDecodingDeletedData 
      hadoop.hdfs.TestReadStripedFileWithMissingBlocks 
      hadoop.hdfs.TestReconstructStripedFile 
      hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy 
      hadoop.hdfs.TestSecureEncryptionZoneWithKMS 
      hadoop.hdfs.TestTrashWithSecureEncryptionZones 
      hadoop.hdfs.tools.TestDebugAdmin 
      hadoop.hdfs.tools.TestDFSAdmin 
      hadoop.hdfs.tools.TestDFSAdminWithHA 
      hadoop.hdfs.tools.TestGetConf 
      hadoop.hdfs.web.TestWebHDFS 
      hadoop.hdfs.web.TestWebHdfsUrl 
      hadoop.fs.http.server.TestHttpFSServerWebServer 
      hadoop.hdfs.nfs.nfs3.TestRpcProgramNfs3 
      hadoop.hdfs.nfs.nfs3.TestWrites 
      hadoop.yarn.logaggregation.filecontroller.ifile.TestLogAggregationIndexFileController

      hadoop.yarn.logaggregation.TestAggregatedLogFormat 
      hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch 
      hadoop.yarn.server.nodemanager.containermanager.linux.privileged.TestPrivilegedOperationExecutor

      hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestCGroupsHandlerImpl

      hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime

      hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestJavaSandboxLinuxContainerRuntime

      hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestAppLogAggregatorImpl

      hadoop.yarn.server.nodemanager.containermanager.TestContainerManager 
      hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService 
      hadoop.yarn.server.nodemanager.TestContainerExecutor 
      hadoop.yarn.server.nodemanager.TestLocalDirsHandlerService 
      hadoop.yarn.server.nodemanager.TestNodeManagerResync 
      hadoop.yarn.server.nodemanager.webapp.TestContainerLogsPage 
      hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer 
      hadoop.yarn.server.timeline.security.TestTimelineAuthenticationFilterForV1 
      hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher 
      hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore 
      hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.TestLeveldbConfigurationStore

      hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler 
      hadoop.yarn.server.resourcemanager.scheduler.constraint.TestPlacementProcessor 
      hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService 
      hadoop.yarn.server.resourcemanager.TestResourceTrackerService 
      hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore 
      hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage

      hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity 
      hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun 
      hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction 
      hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps 
      hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities 
      hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema 
      hadoop.yarn.applications.distributedshell.TestDistributedShell 
      hadoop.mapred.TestTaskProgressReporter 
      hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter 
      hadoop.mapreduce.lib.output.TestFileOutputCommitter 
      hadoop.mapred.TestJavaSerialization 
      hadoop.mapred.TestLocalJobSubmission 
      hadoop.mapred.TestMiniMRChildTask 
      hadoop.mapred.TestMRTimelineEventHandling 
      hadoop.mapreduce.v2.TestMRJobs 
      hadoop.mapreduce.v2.TestUberAM 
      hadoop.yarn.service.client.TestBuildExternalComponents 
      hadoop.yarn.service.client.TestServiceCLI 
      hadoop.yarn.service.monitor.TestServiceMonitor 
      hadoop.yarn.service.providers.TestAbstractClientProvider 
      hadoop.yarn.service.TestServiceAM 
      hadoop.yarn.service.TestYarnNativeServices 
      hadoop.mapred.nativetask.handlers.TestNativeCollectorOnlyHandler 
      hadoop.mapred.uploader.TestFrameworkUploader 
      hadoop.tools.TestDistCpSystem 
      hadoop.tools.TestHadoopArchiveLogs 
      hadoop.mapred.gridmix.TestGridMixClasses 
      hadoop.fs.azure.TestClientThrottlingAnalyzer 
      hadoop.yarn.sls.TestReservationSystemInvariants 


  CTEST:

      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt>
 [28K]

  unit:

      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt>
 [276K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt>
 [1.5M]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt>
 [68K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt>
 [20K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-nfs.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-nfs.txt>
 [16K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt>
 [40K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt>
 [216K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt>
 [32K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt>
 [116K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt>
 [12K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt>
 [12K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt>
 [20K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt>
[136K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt>
 [108K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.txt>
 [92K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt>
 [12K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-uploader.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-uploader.txt>
 [20K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt>
 [16K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-tools_hadoop-archive-logs.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-tools_hadoop-archive-logs.txt>
 [8.0K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-tools_hadoop-gridmix.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-tools_hadoop-gridmix.txt>
 [12K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt>
 [16K]
      https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
<https://builds.apache.org/job/hadoop-trunk-win/406/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt>
 [12K]

Powered by Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org <http://yetus.apache.org/>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message