hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sunil G <sun...@apache.org>
Subject Re: Hadoop 3.2 Release Plan proposal
Date Mon, 17 Sep 2018 09:51:43 GMT
Hi All

We are down to 3 Blockers and 4 Critical now. Thanks all of you for helping
in this. I am following up on these tickets, once its closed we will cut
the 3.2 branch.

Thanks
Sunil Govindan

On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org> wrote:

> Hi All,
>
> Inline with the original 3.2 communication proposal dated 17th July 2018,
> I would like to provide more updates.
>
> We are approaching previously proposed code freeze date (September 14,
> 2018). So I would like to cut 3.2 branch on 17th Sept and point existing
> trunk to 3.3 if there are no issues.
>
> *Current Release Plan:*
> Feature freeze date : all features to merge by September 7, 2018.
> Code freeze date : blockers/critical only, no improvements and
> blocker/critical bug-fixes September 14, 2018.
> Release date: September 28, 2018
>
> If any critical/blocker tickets which are targeted to 3.2.0, we need to
> backport to 3.2 post branch cut.
>
> Here's an updated 3.2.0 feature status:
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI.
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
>
> 2. Features close to finish:
>
> - (Steve) S3Guard Phase III. Close to commit.
> - (Steve) S3a phase V. Close to commit.
> - (Steve) Support Windows Azure Storage. Close to commit.
>
> 3. Tentative/Cancelled features for 3.2:
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
> 19 Blocker and Critical issues [1] are open, I am following up with owners
> to get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
>
>
> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <sunilg@apache.org> wrote:
>
>> Hi All,
>>
>> Inline with earlier communication dated 17th July 2018, I would like to
>> provide some updates.
>>
>> We are approaching previously proposed code freeze date (Aug 31).
>>
>> One of the critical feature Node Attributes feature merge discussion/vote
>> is ongoing. Also few other Blocker bugs need a bit more time. With regard
>> to this, suggesting to push the feature/code freeze for 2 more weeks to
>> accommodate these jiras too.
>>
>> Proposing Updated changes in plan inline with this:
>> Feature freeze date : all features to merge by September 7, 2018.
>> Code freeze date : blockers/critical only, no improvements and
>>  blocker/critical bug-fixes September 14, 2018.
>> Release date: September 28, 2018
>>
>> If any features in branch which are targeted to 3.2.0, please reply to
>> this email thread.
>>
>> *Here's an updated 3.2.0 feature status:*
>>
>> 1. Merged & Completed features:
>>
>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
>> Initial cut.
>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>> and CLI.
>>
>> 2. Features close to finish:
>>
>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
>> Ongoing.
>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> ATSv2. Patch in progress.
>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
>> In progress.
>>
>> 3. Tentative features:
>>
>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>> be done before Aug 2018.
>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> Challenging as more discussions are on-going.
>>
>> *Summary of 3.2.0 issues status:*
>>
>> 26 Blocker and Critical issues [1] are open, I am following up with
>> owners to get status on each of them to get in by Code Freeze date.
>>
>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>> BY priority DESC
>>
>> Thanks,
>> Sunil
>>
>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <sunilg@apache.org> wrote:
>>
>>> Hi All,
>>>
>>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>>> July 2018, I would like to provide some updates.
>>>
>>> We are approaching previously proposed feature freeze date (Aug 21,
>>> about 7 days from today).
>>> If any features in branch which are targeted to 3.2.0, please reply to
>>> this email thread.
>>> Steve has mentioned about the s3 features which will come close to Code
>>> Freeze Date (Aug 31st).
>>>
>>> *Here's an updated 3.2.0 feature status:*
>>>
>>> 1. Merged & Completed features:
>>>
>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> workloads Initial cut.
>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>>
>>> 2. Features close to finish:
>>>
>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
>>> are all in, only one last
>>> patch is in review state.
>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> Scheduler. Close to commit.
>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>>> and CLI. 2 patches are pending
>>> which will be closed by Feature freeze date.
>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>> ATSv2. Patch in progress.
>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
>>> In progress.
>>>
>>> 3. Tentative features:
>>>
>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>> be done before Aug 2018.
>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> Challenging as more discussions are on-going.
>>>
>>> *Summary of 3.2.0 issues status:*
>>>
>>> 39 Blocker and Critical issues [1] are open, I am checking with owners
>>> to get status on each of them to get in by Code Freeze date.
>>>
>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>>> BY priority DESC
>>>
>>> Thanks,
>>> Sunil
>>>
>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <sunilg@apache.org> wrote:
>>>
>>>> Thanks Subru for the thoughts.
>>>> One of the main reason for a major release is to push out critical
>>>> features with a faster cadence to the users. If we are pulling more and
>>>> more different types of features to a minor release, that branch will
>>>> become more destabilized and it may be tough to say that 3.1.2 is stable
>>>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>>>> subsequent minor release.
>>>> For few companies, it makes sense to push out these new features faster
>>>> to make a reach to the users. Adding to the point to the backporting
>>>> issues, I agree that its a pain and we can workaround that with some git
>>>> scripts. If we can make such scripts available to committers, backport will
>>>> be seem-less across branches and we can achieve the faster release cadence
>>>> also.
>>>>
>>>> Thoughts?
>>>>
>>>> - Sunil
>>>>
>>>>
>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <subru@apache.org>
>>>> wrote:
>>>>
>>>>> Thanks Sunil for volunteering to lead the release effort. I am
>>>>> generally
>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>>>> already
>>>>> have too many branches to be maintained. I already see many commits
>>>>> are in
>>>>> different branches with no apparent rationale, for e.g: 3.1 has commits
>>>>> which are absent in 3.0 etc.
>>>>>
>>>>> Additionally AFAIK 3.x has not been deployed in any major production
>>>>> setting so the cost of adding features should be minimal.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> -Subru
>>>>>
>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <sunilg@apache.org> wrote:
>>>>>
>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>>>> >
>>>>> > Yes, important changes and features are much needed, hence we will
be
>>>>> > keeping the door open for them as possible. Also considering few
more
>>>>> > offline requests from other folks, I think extending the timeframe
by
>>>>> > couple of weeks makes sense (including a second RC buffer) and this
>>>>> should
>>>>> > ideally help us to ship this by September itself.
>>>>> >
>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>>>> >
>>>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>>>> >
>>>>> > - Code freeze date : blockers/critical only, no improvements and
non
>>>>> > blocker/critical
>>>>> >
>>>>> > bug-fixes  August 31, 2018.
>>>>> >
>>>>> > - Release date: September 15, 2018
>>>>> >
>>>>> > Thank Eric and Zian, I think Wangda has already answered your
>>>>> questions.
>>>>> >
>>>>> > Thanks
>>>>> > Sunil
>>>>> >
>>>>> >
>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wheeleast@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
that.
>>>>> > >
>>>>> > > To concerns from Steve,
>>>>> > >
>>>>> > > It is a good idea to keep the door open to get important changes
/
>>>>> > > features in before cutoff. I would prefer to keep the proposed
>>>>> release
>>>>> > date
>>>>> > > to make sure things can happen earlier instead of last minute
and
>>>>> we all
>>>>> > > know that releases are always get delayed :). I'm also fine
if we
>>>>> want
>>>>> > get
>>>>> > > another several weeks time.
>>>>> > >
>>>>> > > Regarding of 3.3 release, I would suggest doing that before
>>>>> thanksgiving.
>>>>> > > Do you think is it good or too early / late?
>>>>> > >
>>>>> > > Eric,
>>>>> > >
>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can
get
>>>>> merged
>>>>> > > in time, we probably not need the YARN-8220.
>>>>> > >
>>>>> > > Sunil,
>>>>> > >
>>>>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
>>>>> > Roadmap
>>>>> > > with proposed plan as well? We can fill feature list first
before
>>>>> getting
>>>>> > > consensus of time.
>>>>> > >
>>>>> > > Thanks,
>>>>> > > Wangda
>>>>> > >
>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>>>> <fabbri@cloudera.com.invalid
>>>>> > >
>>>>> > > wrote:
>>>>> > >
>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>>>> stevel@hortonworks.com>
>>>>> > >> wrote:
>>>>> > >>
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>>>> > >> > sunilg@apache.org>> wrote:
>>>>> > >> >
>>>>> > >> > I would also would like to take this opportunity to
come up
>>>>> with a
>>>>> > >> detailed
>>>>> > >> > plan.
>>>>> > >> >
>>>>> > >> > - Feature freeze date : all features should be merged
by August
>>>>> 10,
>>>>> > >> 2018.
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > <snip>
>>>>> > >>
>>>>> > >> >
>>>>> > >> > Please let me know if I missed any features targeted
to 3.2 per
>>>>> this
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>>>> > >> >
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake
Gen 2
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>>>> > >> >
>>>>> > >> > I don't think this is quite ready yet, I've been doing
work on
>>>>> it, but
>>>>> > >> if
>>>>> > >> > we have a 3 week deadline, I'm going to expect some
timely
>>>>> reviews on
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>>>> > >> >
>>>>> > >> > I've uprated that to a blocker feature; will review
the S3 &
>>>>> S3Guard
>>>>> > >> JIRAs
>>>>> > >> > to see which of those are blocking. Then there are
some pressing
>>>>> > "guave,
>>>>> > >> > java 9 prep"
>>>>> > >> >
>>>>> > >> >
>>>>> > >>  I can help with this part if you like.
>>>>> > >>
>>>>> > >>
>>>>> > >>
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > timeline. I would like to volunteer myself as release
manager
>>>>> of 3.2.0
>>>>> > >> > release.
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > well volunteered!
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> Yes, thank you for stepping up.
>>>>> > >>
>>>>> > >>
>>>>> > >> >
>>>>> > >> > I think this raises a good q: what timetable should
we have for
>>>>> the
>>>>> > >> 3.2. &
>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
having the
>>>>> outline
>>>>> > >> time
>>>>> > >> > from the 3.2 to the 3.3 release means that there's
less concern
>>>>> about
>>>>> > >> > things not making the 3.2 dealine
>>>>> > >> >
>>>>> > >> > -Steve
>>>>> > >> >
>>>>> > >> >
>>>>> > >> Good idea to mitigate the short deadline.
>>>>> > >>
>>>>> > >> -AF
>>>>> > >>
>>>>> > >
>>>>> >
>>>>>
>>>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message