hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sunil G <sun...@apache.org>
Subject Re: Hadoop 3.2 Release Plan proposal
Date Thu, 30 Aug 2018 16:29:57 GMT
Hi All,

Inline with earlier communication dated 17th July 2018, I would like to
provide some updates.

We are approaching previously proposed code freeze date (Aug 31).

One of the critical feature Node Attributes feature merge discussion/vote
is ongoing. Also few other Blocker bugs need a bit more time. With regard
to this, suggesting to push the feature/code freeze for 2 more weeks to
accommodate these jiras too.

Proposing Updated changes in plan inline with this:
Feature freeze date : all features to merge by September 7, 2018.
Code freeze date : blockers/critical only, no improvements and
 blocker/critical bug-fixes September 14, 2018.
Release date: September 28, 2018

If any features in branch which are targeted to 3.2.0, please reply to this
email thread.

*Here's an updated 3.2.0 feature status:*

1. Merged & Completed features:

- (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
Initial cut.
- (Uma) HDFS-10285: HDFS Storage Policy Satisfier
- (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
- (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
and CLI.

2. Features close to finish:

- (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
Ongoing.
- (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
Patch in progress.
- (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
- (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage. In
progress.

3. Tentative features:

- (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
done before Aug 2018.
- (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
as more discussions are on-going.

*Summary of 3.2.0 issues status:*

26 Blocker and Critical issues [1] are open, I am following up with owners
to get status on each of them to get in by Code Freeze date.

[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
BY priority DESC

Thanks,
Sunil

On Tue, Aug 14, 2018 at 10:30 PM Sunil G <sunilg@apache.org> wrote:

> Hi All,
>
> Thanks for the feedbacks. Inline with earlier communication dated 17th
> July 2018, I would like to provide some updates.
>
> We are approaching previously proposed feature freeze date (Aug 21, about
> 7 days from today).
> If any features in branch which are targeted to 3.2.0, please reply to
> this email thread.
> Steve has mentioned about the s3 features which will come close to Code
> Freeze Date (Aug 31st).
>
> *Here's an updated 3.2.0 feature status:*
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>
> 2. Features close to finish:
>
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
> are all in, only one last
> patch is in review state.
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> Close to commit.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI. 2 patches are pending
> which will be closed by Feature freeze date.
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> In progress.
>
> 3. Tentative features:
>
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
>
> 39 Blocker and Critical issues [1] are open, I am checking with owners to
> get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <sunilg@apache.org> wrote:
>
>> Thanks Subru for the thoughts.
>> One of the main reason for a major release is to push out critical
>> features with a faster cadence to the users. If we are pulling more and
>> more different types of features to a minor release, that branch will
>> become more destabilized and it may be tough to say that 3.1.2 is stable
>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>> subsequent minor release.
>> For few companies, it makes sense to push out these new features faster
>> to make a reach to the users. Adding to the point to the backporting
>> issues, I agree that its a pain and we can workaround that with some git
>> scripts. If we can make such scripts available to committers, backport will
>> be seem-less across branches and we can achieve the faster release cadence
>> also.
>>
>> Thoughts?
>>
>> - Sunil
>>
>>
>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <subru@apache.org> wrote:
>>
>>> Thanks Sunil for volunteering to lead the release effort. I am generally
>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>> already
>>> have too many branches to be maintained. I already see many commits are
>>> in
>>> different branches with no apparent rationale, for e.g: 3.1 has commits
>>> which are absent in 3.0 etc.
>>>
>>> Additionally AFAIK 3.x has not been deployed in any major production
>>> setting so the cost of adding features should be minimal.
>>>
>>> Thoughts?
>>>
>>> -Subru
>>>
>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <sunilg@apache.org> wrote:
>>>
>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>> >
>>> > Yes, important changes and features are much needed, hence we will be
>>> > keeping the door open for them as possible. Also considering few more
>>> > offline requests from other folks, I think extending the timeframe by
>>> > couple of weeks makes sense (including a second RC buffer) and this
>>> should
>>> > ideally help us to ship this by September itself.
>>> >
>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>> >
>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>> >
>>> > - Code freeze date : blockers/critical only, no improvements and non
>>> > blocker/critical
>>> >
>>> > bug-fixes  August 31, 2018.
>>> >
>>> > - Release date: September 15, 2018
>>> >
>>> > Thank Eric and Zian, I think Wangda has already answered your
>>> questions.
>>> >
>>> > Thanks
>>> > Sunil
>>> >
>>> >
>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wheeleast@gmail.com>
>>> wrote:
>>> >
>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
>>> > >
>>> > > To concerns from Steve,
>>> > >
>>> > > It is a good idea to keep the door open to get important changes /
>>> > > features in before cutoff. I would prefer to keep the proposed
>>> release
>>> > date
>>> > > to make sure things can happen earlier instead of last minute and we
>>> all
>>> > > know that releases are always get delayed :). I'm also fine if we
>>> want
>>> > get
>>> > > another several weeks time.
>>> > >
>>> > > Regarding of 3.3 release, I would suggest doing that before
>>> thanksgiving.
>>> > > Do you think is it good or too early / late?
>>> > >
>>> > > Eric,
>>> > >
>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
>>> merged
>>> > > in time, we probably not need the YARN-8220.
>>> > >
>>> > > Sunil,
>>> > >
>>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
>>> > Roadmap
>>> > > with proposed plan as well? We can fill feature list first before
>>> getting
>>> > > consensus of time.
>>> > >
>>> > > Thanks,
>>> > > Wangda
>>> > >
>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>> <fabbri@cloudera.com.invalid
>>> > >
>>> > > wrote:
>>> > >
>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>> stevel@hortonworks.com>
>>> > >> wrote:
>>> > >>
>>> > >> >
>>> > >> >
>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>> > >> > sunilg@apache.org>> wrote:
>>> > >> >
>>> > >> > I would also would like to take this opportunity to come up
with a
>>> > >> detailed
>>> > >> > plan.
>>> > >> >
>>> > >> > - Feature freeze date : all features should be merged by August
>>> 10,
>>> > >> 2018.
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > <snip>
>>> > >>
>>> > >> >
>>> > >> > Please let me know if I missed any features targeted to 3.2
per
>>> this
>>> > >> >
>>> > >> >
>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>> > >> >
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>> > >> >
>>> > >> >
>>> > >> > There's a bigger bit of work coming on for Azure Datalake
Gen 2
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>> > >> >
>>> > >> > I don't think this is quite ready yet, I've been doing work
on
>>> it, but
>>> > >> if
>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>> reviews on
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>> > >> >
>>> > >> > I've uprated that to a blocker feature; will review the S3
&
>>> S3Guard
>>> > >> JIRAs
>>> > >> > to see which of those are blocking. Then there are some pressing
>>> > "guave,
>>> > >> > java 9 prep"
>>> > >> >
>>> > >> >
>>> > >>  I can help with this part if you like.
>>> > >>
>>> > >>
>>> > >>
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > timeline. I would like to volunteer myself as release manager
of
>>> 3.2.0
>>> > >> > release.
>>> > >> >
>>> > >> >
>>> > >> > well volunteered!
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> Yes, thank you for stepping up.
>>> > >>
>>> > >>
>>> > >> >
>>> > >> > I think this raises a good q: what timetable should we have
for
>>> the
>>> > >> 3.2. &
>>> > >> > 3.3 releases; if we do want a faster cadence, then having
the
>>> outline
>>> > >> time
>>> > >> > from the 3.2 to the 3.3 release means that there's less concern
>>> about
>>> > >> > things not making the 3.2 dealine
>>> > >> >
>>> > >> > -Steve
>>> > >> >
>>> > >> >
>>> > >> Good idea to mitigate the short deadline.
>>> > >>
>>> > >> -AF
>>> > >>
>>> > >
>>> >
>>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message