hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Wang <andrew.w...@cloudera.com>
Subject 2017-09-01 Hadoop 3 release status update
Date Fri, 01 Sep 2017 21:53:29 GMT


We're two weeks out from beta1, focus is on blocker burndown.


   - S3Guard merged!
   - TSv2 alpha2 merged!
   - branch-3.0 has been cut after discussion on dev lists.

Red flags:

   - 10 blockers on the dashboard, closed and bumped some but new ones
   - Still need to land YARN native services and fix some S3Guard doc
   issues for beta1.
   - Rolling upgrade JIRAs for YARN and HDFS are not making any visible

Previously tracked beta1 blockers that have been resolved:

   - HADOOP-13363 <https://issues.apache.org/jira/browse/HADOOP-13363> (Upgrade
   to protobuf 3): I dropped this from beta1 since it's simply not going to
   happen in time.
   - YARN-7076 <https://issues.apache.org/jira/browse/YARN-7076>: This was
   quickly resolved! Thanks Jian, Junping, Jason for the action.
   - YARN-7094 <https://issues.apache.org/jira/browse/YARN-7094> (Document
   that server-side graceful decom is currently not recommended): Patch

beta1 blockers:

   - HADOOP-14826 <https://issues.apache.org/jira/browse/HADOOP-14826> (review
   S3 docs prior to 3.0.0-beta1): New blocker with S3Guard merged. Should just
   be a quick doc update.
   - HADOOP-14284 <https://issues.apache.org/jira/browse/HADOOP-14284> (Shade
   Guava everywhere): Agreement to shade yarn-client at at HADOOP-14771.
   Shading hadoop-hdfs is still being discussed?
   - HADOOP-14771 <https://issues.apache.org/jira/browse/HADOOP-14771>
   does not include hadoop-yarn-client): Patch up, needs review, waiting on
   - YARN-5536 <https://issues.apache.org/jira/browse/YARN-5536> (Multiple
   format support (JSON, etc.) for exclude node file in NM graceful
   decommission with timeout): We're waiting on input from Junping.
   - MAPREDUCE-6941 (The default setting doesn't work for MapReduce job):
   Ray thinks this is a Won't Fix, waiting on Junping to confirm.
   - HADOOP-14238 (Rechecking Guava's object is not exposed to user-facing
   API): This relates to HADOOP-14771, I left a JIRA comment.

beta1 features:

   - Erasure coding
      - There are three must-dos. Two have patches, one might not be a
      - HDFS-11882 has been revved and reviewed, seems close
      - HDFS-11467 and HDFS-7859 are related, Sammi/Eddy/Kai are
      discussing, Sammi thinks we can still make beta1.
   - Addressing incompatible changes (YARN-6142 and HDFS-11096)
      - Sean has HDFS rolling upgrade scripts up, waiting on Ray to add
      some YARN/MR coverage too.
      - Need to do a final runthrough of the JACC reports for YARN and HDFS.
   - Classpath isolation (HADOOP-11656)
      - Sean has retriaged the subtasks and has been posting patches.
   - Compat guide (HADOOP-13714
      - New patch is up, but needs review. Daniel asked Chris Douglas and
      Steve Loughran.
   - YARN native services
      - Jian sent out the merge vote
   - TSv2 alpha 2
   - This was merged, no problems thus far [image: (smile)]

GA features:

   - Resource profiles (Wangda Tan)
      - Merge vote was sent out. Since branch-3.0 has been cut, this can be
      merged to trunk (3.1.0) and then backported once we've completed testing.
   - HDFS router-based federation (Chris Douglas)
   - This is like YARN federation, very separate and doesn't add new APIs,
      run in production at MSFT.
      - If it passes Cloudera internal integration testing, I'm fine
      putting this in for GA.
   - API-based scheduler configuration (Jonathan Hung)
      - Jonathan mentioned that his main goal is to get this in for 2.9.0,
      which seems likely to go out after 3.0.0 GA since there hasn't been any
      serious release planning yet. Jonathan said that delaying this
until 3.1.0
      is fine.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message