hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kumar Vavilapalli <vino...@apache.org>
Subject Re: [VOTE] Release Apache Hadoop 3.0.0 RC0
Date Tue, 21 Nov 2017 01:26:38 GMT
Thanks for all the push, Andrew!

Looking at the RC. Went through my usual check-list. Here's my summary. Will cast my final
vote after comparing and validating my findings with others.

Verification

 - [Check] Successful recompilation from source tar-ball
 - [Check] Signature verification
 - [Check] Generating dist tarballs from source tar-ball
 - [Check] Testing
    -- Start NN, DN, RM, NM, JHS, Timeline Service
    -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, pi
    -- Tested CLIs to print nodes, apps etc and also navigated UIs

Issues found during testing

Major
 - The previously supported way of being able to use different tar-balls for different sub-modules
is completely broken - common and HDFS tar.gz are completely empty.
 - Cannot enable new UI in YARN because it is under a non-default compilation flag. It should
be on by default.
 - One decommissioned node in YARN ResourceManager UI always appears to start with, even when
there are no NodeManagers that are started yet:  Info <IP>:-1, DECOMMISSIONED, null
rack. It shows up only in the UI though, not in the CLI node -list

Minor
 - resourcemanager-metrics.out is going into current directory instead of log directory
 - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even work. Not just deprecated
in favor of timelineserver as was advertised.
 - Spurious warnings on CLI
            17/11/20 17:07:34 INFO conf.Configuration: resource-types.xml not found
            17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.

Side notes

 - When did we stop putting CHANGES files into the source artifacts?
 - Even after "mvn install"ing once, shading is repeated again and again for every new 'mvn
install' even though there are no source changes - we should see how this can be avoided.
 - Compatibility notes
    -- NM's env list is curtailed unlike in 2.x (For e.g, HADOOP_MAPRED_HOME is not automatically
inherited. Correct behavior)
    -- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar into hadoop-mapreduce-client-jobclient-3.0.0-tests.jar

Thanks
+Vinod

> On Nov 14, 2017, at 1:34 PM, Andrew Wang <andrew.wang@cloudera.com> wrote:
> 
> Hi folks,
> 
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
> 
> http://people.apache.org/~wang/3.0.0-RC0/
> 
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> 
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
> 
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
> 
> Best,
> Andrew


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message