hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sunil G <sun...@apache.org>
Subject Re: Hadoop 3.2 Release Plan proposal
Date Wed, 03 Oct 2018 02:13:47 GMT
Thanks Robert and Haibo for quickly correcting same.
Sigh, I somehow missed one file while committing the change. Sorry for the
trouble.

- Sunil

On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rkanter@cloudera.com> wrote:

> Looks like there's two that weren't updated:
> >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
> --include=pom.xml
> ./hadoop-project/pom.xml:
> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
> ./pom.xml:    <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>
> I've just pushed in an addendum commit to fix those.
> In the future, please make sure to do a sanity compile when updating poms.
>
> thanks
> - Robert
>
> On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri <fabbri@cloudera.com.invalid>
> wrote:
>
>> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>> top-level pom.xml?
>>
>>
>> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org> wrote:
>>
>> > Hi All
>> >
>> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
>> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
>> > patches are pulled into branch-3.2.
>> >
>> > Thank You
>> > - Sunil
>> >
>> >
>> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org> wrote:
>> >
>> > > Hi All
>> > >
>> > > We are now down to the last Blocker and HADOOP-15407 is merged to
>> trunk.
>> > > Thanks for the support.
>> > >
>> > > *Plan for RC*
>> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>> > > RC0 for 3.2: *28th Friday*
>> > >
>> > > Thank You
>> > > Sunil
>> > >
>> > >
>> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org> wrote:
>> > >
>> > >> Hi All
>> > >>
>> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
>> > >> helping in this. I am following up on these tickets, once its closed
>> we
>> > >> will cut the 3.2 branch.
>> > >>
>> > >> Thanks
>> > >> Sunil Govindan
>> > >>
>> > >>
>> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org> wrote:
>> > >>
>> > >>> Hi All,
>> > >>>
>> > >>> Inline with the original 3.2 communication proposal dated 17th
July
>> > >>> 2018, I would like to provide more updates.
>> > >>>
>> > >>> We are approaching previously proposed code freeze date (September
>> 14,
>> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
>> > existing
>> > >>> trunk to 3.3 if there are no issues.
>> > >>>
>> > >>> *Current Release Plan:*
>> > >>> Feature freeze date : all features to merge by September 7, 2018.
>> > >>> Code freeze date : blockers/critical only, no improvements and
>> > >>> blocker/critical bug-fixes September 14, 2018.
>> > >>> Release date: September 28, 2018
>> > >>>
>> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
>> need to
>> > >>> backport to 3.2 post branch cut.
>> > >>>
>> > >>> Here's an updated 3.2.0 feature status:
>> > >>>
>> > >>> 1. Merged & Completed features:
>> > >>>
>> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> > >>> workloads Initial cut.
>> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> > >>> Scheduler.
>> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
>> > API
>> > >>> and CLI.
>> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
>> works.
>> > >>>
>> > >>> 2. Features close to finish:
>> > >>>
>> > >>> - (Steve) S3Guard Phase III. Close to commit.
>> > >>> - (Steve) S3a phase V. Close to commit.
>> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>> > >>>
>> > >>> 3. Tentative/Cancelled features for 3.2:
>> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
from
>> > >>> ATSv2. Patch in progress.
>> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> challenging to
>> > >>> be done before Aug 2018.
>> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> > >>> Challenging as more discussions are on-going.
>> > >>>
>> > >>> *Summary of 3.2.0 issues status:*
>> > >>> 19 Blocker and Critical issues [1] are open, I am following up
with
>> > >>> owners to get status on each of them to get in by Code Freeze date.
>> > >>>
>> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> > (Blocker,
>> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
3.2.0
>> > ORDER
>> > >>> BY priority DESC
>> > >>>
>> > >>> Thanks,
>> > >>> Sunil
>> > >>>
>> > >>>
>> > >>>
>> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <sunilg@apache.org>
wrote:
>> > >>>
>> > >>>> Hi All,
>> > >>>>
>> > >>>> Inline with earlier communication dated 17th July 2018, I would
>> like
>> > to
>> > >>>> provide some updates.
>> > >>>>
>> > >>>> We are approaching previously proposed code freeze date (Aug
31).
>> > >>>>
>> > >>>> One of the critical feature Node Attributes feature merge
>> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need
a bit
>> > more
>> > >>>> time. With regard to this, suggesting to push the feature/code
>> freeze
>> > for 2
>> > >>>> more weeks to accommodate these jiras too.
>> > >>>>
>> > >>>> Proposing Updated changes in plan inline with this:
>> > >>>> Feature freeze date : all features to merge by September 7,
2018.
>> > >>>> Code freeze date : blockers/critical only, no improvements
and
>> > >>>>  blocker/critical bug-fixes September 14, 2018.
>> > >>>> Release date: September 28, 2018
>> > >>>>
>> > >>>> If any features in branch which are targeted to 3.2.0, please
>> reply to
>> > >>>> this email thread.
>> > >>>>
>> > >>>> *Here's an updated 3.2.0 feature status:*
>> > >>>>
>> > >>>> 1. Merged & Completed features:
>> > >>>>
>> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> > >>>> workloads Initial cut.
>> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> > >>>> Scheduler.
>> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> Service
>> > >>>> API and CLI.
>> > >>>>
>> > >>>> 2. Features close to finish:
>> > >>>>
>> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> Merge/Vote
>> > >>>> Ongoing.
>> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
from
>> > >>>> ATSv2. Patch in progress.
>> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
>> > works.
>> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>> > >>>> Storage. In progress.
>> > >>>>
>> > >>>> 3. Tentative features:
>> > >>>>
>> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> challenging
>> > to
>> > >>>> be done before Aug 2018.
>> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> > >>>> Challenging as more discussions are on-going.
>> > >>>>
>> > >>>> *Summary of 3.2.0 issues status:*
>> > >>>>
>> > >>>> 26 Blocker and Critical issues [1] are open, I am following
up with
>> > >>>> owners to get status on each of them to get in by Code Freeze
date.
>> > >>>>
>> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
in
>> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> > Version/s" =
>> > >>>> 3.2.0 ORDER BY priority DESC
>> > >>>>
>> > >>>> Thanks,
>> > >>>> Sunil
>> > >>>>
>> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <sunilg@apache.org>
>> wrote:
>> > >>>>
>> > >>>>> Hi All,
>> > >>>>>
>> > >>>>> Thanks for the feedbacks. Inline with earlier communication
dated
>> > 17th
>> > >>>>> July 2018, I would like to provide some updates.
>> > >>>>>
>> > >>>>> We are approaching previously proposed feature freeze date
(Aug
>> 21,
>> > >>>>> about 7 days from today).
>> > >>>>> If any features in branch which are targeted to 3.2.0,
please
>> reply
>> > to
>> > >>>>> this email thread.
>> > >>>>> Steve has mentioned about the s3 features which will come
close to
>> > >>>>> Code Freeze Date (Aug 31st).
>> > >>>>>
>> > >>>>> *Here's an updated 3.2.0 feature status:*
>> > >>>>>
>> > >>>>> 1. Merged & Completed features:
>> > >>>>>
>> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> > >>>>> workloads Initial cut.
>> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> > >>>>>
>> > >>>>> 2. Features close to finish:
>> > >>>>>
>> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
Major
>> > >>>>> patches are all in, only one last
>> > >>>>> patch is in review state.
>> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> > >>>>> Scheduler. Close to commit.
>> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
YARN
>> Service
>> > >>>>> API and CLI. 2 patches are pending
>> > >>>>> which will be closed by Feature freeze date.
>> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
apps
>> from
>> > >>>>> ATSv2. Patch in progress.
>> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
>> > >>>>> works.
>> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
Azure
>> > >>>>> Storage. In progress.
>> > >>>>>
>> > >>>>> 3. Tentative features:
>> > >>>>>
>> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> challenging
>> > >>>>> to be done before Aug 2018.
>> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> > >>>>> Challenging as more discussions are on-going.
>> > >>>>>
>> > >>>>> *Summary of 3.2.0 issues status:*
>> > >>>>>
>> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
with
>> > owners
>> > >>>>> to get status on each of them to get in by Code Freeze
date.
>> > >>>>>
>> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
in
>> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> > Version/s" =
>> > >>>>> 3.2.0 ORDER BY priority DESC
>> > >>>>>
>> > >>>>> Thanks,
>> > >>>>> Sunil
>> > >>>>>
>> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <sunilg@apache.org>
>> wrote:
>> > >>>>>
>> > >>>>>> Thanks Subru for the thoughts.
>> > >>>>>> One of the main reason for a major release is to push
out
>> critical
>> > >>>>>> features with a faster cadence to the users. If we
are pulling
>> more
>> > and
>> > >>>>>> more different types of features to a minor release,
that branch
>> > will
>> > >>>>>> become more destabilized and it may be tough to say
that 3.1.2 is
>> > stable
>> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>> features
>> > in
>> > >>>>>> subsequent minor release.
>> > >>>>>> For few companies, it makes sense to push out these
new features
>> > >>>>>> faster to make a reach to the users. Adding to the
point to the
>> > backporting
>> > >>>>>> issues, I agree that its a pain and we can workaround
that with
>> > some git
>> > >>>>>> scripts. If we can make such scripts available to committers,
>> > backport will
>> > >>>>>> be seem-less across branches and we can achieve the
faster
>> release
>> > cadence
>> > >>>>>> also.
>> > >>>>>>
>> > >>>>>> Thoughts?
>> > >>>>>>
>> > >>>>>> - Sunil
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <subru@apache.org
>> >
>> > >>>>>> wrote:
>> > >>>>>>
>> > >>>>>>> Thanks Sunil for volunteering to lead the release
effort. I am
>> > >>>>>>> generally
>> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer
a 3.1.x) as
>> feel we
>> > >>>>>>> already
>> > >>>>>>> have too many branches to be maintained. I already
see many
>> commits
>> > >>>>>>> are in
>> > >>>>>>> different branches with no apparent rationale,
for e.g: 3.1 has
>> > >>>>>>> commits
>> > >>>>>>> which are absent in 3.0 etc.
>> > >>>>>>>
>> > >>>>>>> Additionally AFAIK 3.x has not been deployed in
any major
>> > production
>> > >>>>>>> setting so the cost of adding features should be
minimal.
>> > >>>>>>>
>> > >>>>>>> Thoughts?
>> > >>>>>>>
>> > >>>>>>> -Subru
>> > >>>>>>>
>> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <sunilg@apache.org>
>> > wrote:
>> > >>>>>>>
>> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>> > >>>>>>> >
>> > >>>>>>> > Yes, important changes and features are much
needed, hence we
>> > will
>> > >>>>>>> be
>> > >>>>>>> > keeping the door open for them as possible.
Also considering
>> few
>> > >>>>>>> more
>> > >>>>>>> > offline requests from other folks, I think
extending the
>> > timeframe
>> > >>>>>>> by
>> > >>>>>>> > couple of weeks makes sense (including a second
RC buffer) and
>> > >>>>>>> this should
>> > >>>>>>> > ideally help us to ship this by September
itself.
>> > >>>>>>> >
>> > >>>>>>> > Revised dates (I will be updating same in
Roadmap wiki as
>> well)
>> > >>>>>>> >
>> > >>>>>>> > - Feature freeze date : all features to merge
by August 21,
>> 2018.
>> > >>>>>>> >
>> > >>>>>>> > - Code freeze date : blockers/critical only,
no improvements
>> and
>> > >>>>>>> non
>> > >>>>>>> > blocker/critical
>> > >>>>>>> >
>> > >>>>>>> > bug-fixes  August 31, 2018.
>> > >>>>>>> >
>> > >>>>>>> > - Release date: September 15, 2018
>> > >>>>>>> >
>> > >>>>>>> > Thank Eric and Zian, I think Wangda has already
answered your
>> > >>>>>>> questions.
>> > >>>>>>> >
>> > >>>>>>> > Thanks
>> > >>>>>>> > Sunil
>> > >>>>>>> >
>> > >>>>>>> >
>> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan
<
>> wheeleast@gmail.com
>> > >
>> > >>>>>>> wrote:
>> > >>>>>>> >
>> > >>>>>>> > > Thanks Sunil for volunteering to be RM
of 3.2 release, +1
>> for
>> > >>>>>>> that.
>> > >>>>>>> > >
>> > >>>>>>> > > To concerns from Steve,
>> > >>>>>>> > >
>> > >>>>>>> > > It is a good idea to keep the door open
to get important
>> > changes
>> > >>>>>>> /
>> > >>>>>>> > > features in before cutoff. I would prefer
to keep the
>> proposed
>> > >>>>>>> release
>> > >>>>>>> > date
>> > >>>>>>> > > to make sure things can happen earlier
instead of last
>> minute
>> > >>>>>>> and we all
>> > >>>>>>> > > know that releases are always get delayed
:). I'm also fine
>> if
>> > >>>>>>> we want
>> > >>>>>>> > get
>> > >>>>>>> > > another several weeks time.
>> > >>>>>>> > >
>> > >>>>>>> > > Regarding of 3.3 release, I would suggest
doing that before
>> > >>>>>>> thanksgiving.
>> > >>>>>>> > > Do you think is it good or too early
/ late?
>> > >>>>>>> > >
>> > >>>>>>> > > Eric,
>> > >>>>>>> > >
>> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135,
if YARN-8135
>> can
>> > >>>>>>> get merged
>> > >>>>>>> > > in time, we probably not need the YARN-8220.
>> > >>>>>>> > >
>> > >>>>>>> > > Sunil,
>> > >>>>>>> > >
>> > >>>>>>> > > Could u update
>> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>> > >>>>>>> > Roadmap
>> > >>>>>>> > > with proposed plan as well? We can fill
feature list first
>> > >>>>>>> before getting
>> > >>>>>>> > > consensus of time.
>> > >>>>>>> > >
>> > >>>>>>> > > Thanks,
>> > >>>>>>> > > Wangda
>> > >>>>>>> > >
>> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron
Fabbri
>> > >>>>>>> <fabbri@cloudera.com.invalid
>> > >>>>>>> > >
>> > >>>>>>> > > wrote:
>> > >>>>>>> > >
>> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve
Loughran <
>> > >>>>>>> stevel@hortonworks.com>
>> > >>>>>>> > >> wrote:
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil
G <sunilg@apache.org
>> > <mailto:
>> > >>>>>>> > >> > sunilg@apache.org>> wrote:
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I would also would like to take
this opportunity to come
>> up
>> > >>>>>>> with a
>> > >>>>>>> > >> detailed
>> > >>>>>>> > >> > plan.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > - Feature freeze date : all
features should be merged by
>> > >>>>>>> August 10,
>> > >>>>>>> > >> 2018.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > <snip>
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > Please let me know if I missed
any features targeted to
>> 3.2
>> > >>>>>>> per this
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > Well there these big todo lists
for S3 & S3Guard.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > There's a bigger bit of work
coming on for Azure Datalake
>> > Gen
>> > >>>>>>> 2
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I don't think this is quite
ready yet, I've been doing
>> work
>> > >>>>>>> on it, but
>> > >>>>>>> > >> if
>> > >>>>>>> > >> > we have a 3 week deadline, I'm
going to expect some
>> timely
>> > >>>>>>> reviews on
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I've uprated that to a blocker
feature; will review the
>> S3 &
>> > >>>>>>> S3Guard
>> > >>>>>>> > >> JIRAs
>> > >>>>>>> > >> > to see which of those are blocking.
Then there are some
>> > >>>>>>> pressing
>> > >>>>>>> > "guave,
>> > >>>>>>> > >> > java 9 prep"
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >>  I can help with this part if you
like.
>> > >>>>>>> > >>
>> > >>>>>>> > >>
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > timeline. I would like to volunteer
myself as release
>> > manager
>> > >>>>>>> of 3.2.0
>> > >>>>>>> > >> > release.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > well volunteered!
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> Yes, thank you for stepping up.
>> > >>>>>>> > >>
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I think this raises a good q:
what timetable should we
>> have
>> > >>>>>>> for the
>> > >>>>>>> > >> 3.2. &
>> > >>>>>>> > >> > 3.3 releases; if we do want
a faster cadence, then having
>> > the
>> > >>>>>>> outline
>> > >>>>>>> > >> time
>> > >>>>>>> > >> > from the 3.2 to the 3.3 release
means that there's less
>> > >>>>>>> concern about
>> > >>>>>>> > >> > things not making the 3.2 dealine
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > -Steve
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> Good idea to mitigate the short deadline.
>> > >>>>>>> > >>
>> > >>>>>>> > >> -AF
>> > >>>>>>> > >>
>> > >>>>>>> > >
>> > >>>>>>> >
>> > >>>>>>>
>> > >>>>>>
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message