hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wangda Tan <wheele...@gmail.com>
Subject Re: [DISCUSS] Merge YARN resource profile (YARN-3926) branch into trunk
Date Fri, 18 Aug 2017 03:27:53 GMT
+hdfs/common/mr

On Thu, Aug 17, 2017 at 1:28 PM, Wangda Tan <wheeleast@gmail.com> wrote:

> Hi all,
>
> I want to hear your thoughts of merging YARN resource profile branch into
> trunk in the next few weeks. The goal is to get it in for Hadoop 3.0 beta1.
>
> *Regarding to testing:*
> We did extensive tests for the feature in the last several months.
> Comparing to latest trunk.
> - For SLS benchmark: We didn't see observable performance gap from
> simulated test based on 8K nodes SLS traces (1 PB memory). We got 3k+
> containers allocated per second.
> - For microbenchmark: We use performance test cases added by YARN-6775, it
> shows around 5% performance regression comparing to trunk.
>
> *Regarding to API stability: *
> Most new added @Public APIs are @Unstable (We're going to convert some new
> added @Public/@Evolving to @Unstable in the cleanup JIRA as well), we want
> to get this included by beta1 so we get some feedbacks before declaring
> stable API.
>
> There're few pending cleanups under YARN-3926 umbrella JIRA. Besides these
> cleanups, this feature works from end-to-end, we will do another iteration
> of end-to-end tests after cleanup patches got committed.
>
> We would love to get your thoughts before opening a voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this efforts including design discussion / patch / reviews, etc.: Varun
> Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu,
> Karthik Kambatla, Jason Lowe, Arun Suresh.
>
> Thanks,
> Wangda Tan
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message