hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Lowe <jl...@yahoo-inc.com.INVALID>
Subject Re: [VOTE] merge feature branch YARN-2928
Date Thu, 07 Jul 2016 16:22:54 GMT
+1 (binding)
Looking forward to seeing this move forward and mature!  I especially appreciate the effort
made to ensure it minimizes the impact to users who are not enabling the feature.  Thank
you to everyone involved!

      From: Sangjin Lee <sjlee@apache.org>
 To: "yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org> 
 Sent: Friday, July 1, 2016 11:00 AM
 Subject: [VOTE] merge feature branch YARN-2928
Hi folks,

Per earlier discussion [1], I'd like to start a formal vote to merge
feature branch YARN-2928 [2] (Timeline Service v.2) to trunk. The vote will
run for 7 days, and will end July 8 9:00 AM PDT.

We have been developing the feature in a feature branch (YARN-2928 [3]) for
a while, and we are reasonably confident that the state of the feature
meets the criteria to be merged onto trunk and we'd love folks to get their
hands on it in a test capacity and provide valuable feedback so that we can
make it production-ready.

In a nutshell, Timeline Service v.2 delivers significant scalability and
usability improvements based on a new architecture. You can browse the
requirements/design doc, the storage schema doc, the new entity/data model,
the YARN documentation, and also discussions on subsequent milestones on
YARN-2928 [2].

What we would like to merge to trunk is termed "alpha 1" (milestone 1). The
feature has a complete end-to-end read/write flow, and you should be able
to start setting it up and testing it. At a high level, the following are
the key features that have been implemented:

- distributed writers (collectors) as NM aux services
- HBase storage
- new entity model that includes flows
- setting the flow context via YARN app tags
- real time metrics aggregation to the application level and the flow level
- rich REST API that supports filters, complex conditionals, limits,
content selection, etc.
- YARN generic events and system metrics
- integration with Distributed Shell and MapReduce

There are a total of 139 subtasks that were completed as part of this

We paid close attention to ensure that once disabled Timeline Service v.2
does not impact existing functionality when disabled (by default).

Special thanks go to all the folks who worked real hard on this feature: Li
Lu, Junping Du, Vinod Kumar Vavilapalli, Naga G. R., Varun Saxena, Vrushali
C, and Joep Rottinghuis!


[2] https://issues.apache.org/jira/browse/YARN-2928
[3] https://github.com/apache/hadoop/commits/YARN-2928

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message