hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Virajith Jalaparti <virajit...@gmail.com>
Subject Re: Hadoop 3.2 Release Plan proposal
Date Thu, 30 Aug 2018 21:52:33 GMT
Hi Sunil,

Quick correction on the task list  (missed this earlier) -- HDFS-12615 is
being done by Inigo Goiri

-Virajith



On Thu, Aug 30, 2018 at 9:30 AM Sunil G <sunilg@apache.org> wrote:

> Hi All,
>
> Inline with earlier communication dated 17th July 2018, I would like to
> provide some updates.
>
> We are approaching previously proposed code freeze date (Aug 31).
>
> One of the critical feature Node Attributes feature merge discussion/vote
> is ongoing. Also few other Blocker bugs need a bit more time. With regard
> to this, suggesting to push the feature/code freeze for 2 more weeks to
> accommodate these jiras too.
>
> Proposing Updated changes in plan inline with this:
> Feature freeze date : all features to merge by September 7, 2018.
> Code freeze date : blockers/critical only, no improvements and
>  blocker/critical bug-fixes September 14, 2018.
> Release date: September 28, 2018
>
> If any features in branch which are targeted to 3.2.0, please reply to this
> email thread.
>
> *Here's an updated 3.2.0 feature status:*
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI.
>
> 2. Features close to finish:
>
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> Ongoing.
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage. In
> progress.
>
> 3. Tentative features:
>
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
>
> 26 Blocker and Critical issues [1] are open, I am following up with owners
> to get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <sunilg@apache.org> wrote:
>
> > Hi All,
> >
> > Thanks for the feedbacks. Inline with earlier communication dated 17th
> > July 2018, I would like to provide some updates.
> >
> > We are approaching previously proposed feature freeze date (Aug 21, about
> > 7 days from today).
> > If any features in branch which are targeted to 3.2.0, please reply to
> > this email thread.
> > Steve has mentioned about the s3 features which will come close to Code
> > Freeze Date (Aug 31st).
> >
> > *Here's an updated 3.2.0 feature status:*
> >
> > 1. Merged & Completed features:
> >
> > - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> > Initial cut.
> > - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >
> > 2. Features close to finish:
> >
> > - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
> > are all in, only one last
> > patch is in review state.
> > - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> > Close to commit.
> > - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> > and CLI. 2 patches are pending
> > which will be closed by Feature freeze date.
> > - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> ATSv2.
> > Patch in progress.
> > - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> > - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> > In progress.
> >
> > 3. Tentative features:
> >
> > - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
> be
> > done before Aug 2018.
> > - (Eric) YARN-7129: Application Catalog for YARN applications.
> Challenging
> > as more discussions are on-going.
> >
> > *Summary of 3.2.0 issues status:*
> >
> > 39 Blocker and Critical issues [1] are open, I am checking with owners to
> > get status on each of them to get in by Code Freeze date.
> >
> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> ORDER
> > BY priority DESC
> >
> > Thanks,
> > Sunil
> >
> > On Fri, Jul 20, 2018 at 8:03 AM Sunil G <sunilg@apache.org> wrote:
> >
> >> Thanks Subru for the thoughts.
> >> One of the main reason for a major release is to push out critical
> >> features with a faster cadence to the users. If we are pulling more and
> >> more different types of features to a minor release, that branch will
> >> become more destabilized and it may be tough to say that 3.1.2 is stable
> >> that 3.1.1 for eg. We always tend to improve and stabilize features in
> >> subsequent minor release.
> >> For few companies, it makes sense to push out these new features faster
> >> to make a reach to the users. Adding to the point to the backporting
> >> issues, I agree that its a pain and we can workaround that with some git
> >> scripts. If we can make such scripts available to committers, backport
> will
> >> be seem-less across branches and we can achieve the faster release
> cadence
> >> also.
> >>
> >> Thoughts?
> >>
> >> - Sunil
> >>
> >>
> >> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <subru@apache.org>
> wrote:
> >>
> >>> Thanks Sunil for volunteering to lead the release effort. I am
> generally
> >>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
> >>> already
> >>> have too many branches to be maintained. I already see many commits are
> >>> in
> >>> different branches with no apparent rationale, for e.g: 3.1 has commits
> >>> which are absent in 3.0 etc.
> >>>
> >>> Additionally AFAIK 3.x has not been deployed in any major production
> >>> setting so the cost of adding features should be minimal.
> >>>
> >>> Thoughts?
> >>>
> >>> -Subru
> >>>
> >>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <sunilg@apache.org> wrote:
> >>>
> >>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>> >
> >>> > Yes, important changes and features are much needed, hence we will
be
> >>> > keeping the door open for them as possible. Also considering few more
> >>> > offline requests from other folks, I think extending the timeframe
by
> >>> > couple of weeks makes sense (including a second RC buffer) and this
> >>> should
> >>> > ideally help us to ship this by September itself.
> >>> >
> >>> > Revised dates (I will be updating same in Roadmap wiki as well)
> >>> >
> >>> > - Feature freeze date : all features to merge by August 21, 2018.
> >>> >
> >>> > - Code freeze date : blockers/critical only, no improvements and non
> >>> > blocker/critical
> >>> >
> >>> > bug-fixes  August 31, 2018.
> >>> >
> >>> > - Release date: September 15, 2018
> >>> >
> >>> > Thank Eric and Zian, I think Wangda has already answered your
> >>> questions.
> >>> >
> >>> > Thanks
> >>> > Sunil
> >>> >
> >>> >
> >>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wheeleast@gmail.com>
> >>> wrote:
> >>> >
> >>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
that.
> >>> > >
> >>> > > To concerns from Steve,
> >>> > >
> >>> > > It is a good idea to keep the door open to get important changes
/
> >>> > > features in before cutoff. I would prefer to keep the proposed
> >>> release
> >>> > date
> >>> > > to make sure things can happen earlier instead of last minute
and
> we
> >>> all
> >>> > > know that releases are always get delayed :). I'm also fine if
we
> >>> want
> >>> > get
> >>> > > another several weeks time.
> >>> > >
> >>> > > Regarding of 3.3 release, I would suggest doing that before
> >>> thanksgiving.
> >>> > > Do you think is it good or too early / late?
> >>> > >
> >>> > > Eric,
> >>> > >
> >>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can
get
> >>> merged
> >>> > > in time, we probably not need the YARN-8220.
> >>> > >
> >>> > > Sunil,
> >>> > >
> >>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
> >>> > Roadmap
> >>> > > with proposed plan as well? We can fill feature list first before
> >>> getting
> >>> > > consensus of time.
> >>> > >
> >>> > > Thanks,
> >>> > > Wangda
> >>> > >
> >>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>> <fabbri@cloudera.com.invalid
> >>> > >
> >>> > > wrote:
> >>> > >
> >>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>> stevel@hortonworks.com>
> >>> > >> wrote:
> >>> > >>
> >>> > >> >
> >>> > >> >
> >>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
> >>> > >> > sunilg@apache.org>> wrote:
> >>> > >> >
> >>> > >> > I would also would like to take this opportunity to come
up
> with a
> >>> > >> detailed
> >>> > >> > plan.
> >>> > >> >
> >>> > >> > - Feature freeze date : all features should be merged
by August
> >>> 10,
> >>> > >> 2018.
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >> > <snip>
> >>> > >>
> >>> > >> >
> >>> > >> > Please let me know if I missed any features targeted
to 3.2 per
> >>> this
> >>> > >> >
> >>> > >> >
> >>> > >> > Well there these big todo lists for S3 & S3Guard.
> >>> > >> >
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >>> > >> >
> >>> > >> >
> >>> > >> > There's a bigger bit of work coming on for Azure Datalake
Gen 2
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >>> > >> >
> >>> > >> > I don't think this is quite ready yet, I've been doing
work on
> >>> it, but
> >>> > >> if
> >>> > >> > we have a 3 week deadline, I'm going to expect some timely
> >>> reviews on
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >>> > >> >
> >>> > >> > I've uprated that to a blocker feature; will review the
S3 &
> >>> S3Guard
> >>> > >> JIRAs
> >>> > >> > to see which of those are blocking. Then there are some
pressing
> >>> > "guave,
> >>> > >> > java 9 prep"
> >>> > >> >
> >>> > >> >
> >>> > >>  I can help with this part if you like.
> >>> > >>
> >>> > >>
> >>> > >>
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >> > timeline. I would like to volunteer myself as release
manager of
> >>> 3.2.0
> >>> > >> > release.
> >>> > >> >
> >>> > >> >
> >>> > >> > well volunteered!
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >> Yes, thank you for stepping up.
> >>> > >>
> >>> > >>
> >>> > >> >
> >>> > >> > I think this raises a good q: what timetable should we
have for
> >>> the
> >>> > >> 3.2. &
> >>> > >> > 3.3 releases; if we do want a faster cadence, then having
the
> >>> outline
> >>> > >> time
> >>> > >> > from the 3.2 to the 3.3 release means that there's less
concern
> >>> about
> >>> > >> > things not making the 3.2 dealine
> >>> > >> >
> >>> > >> > -Steve
> >>> > >> >
> >>> > >> >
> >>> > >> Good idea to mitigate the short deadline.
> >>> > >>
> >>> > >> -AF
> >>> > >>
> >>> > >
> >>> >
> >>>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message