hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wei-Chiu Chuang <weic...@cloudera.com.INVALID>
Subject Re: [DISCUSS] Hadoop 3.3.1 release
Date Thu, 13 May 2021 01:04:48 GMT
Hello it's me again.

While working on the hadoop-thirdparty release, I would like to make
progress on the Hadoop 3.3.1 release in parallel as well.
Looking at the jira dashboard, I think the only real blocker is HDFS-15790
https://github.com/apache/hadoop/pull/2767

I'll try to review the remaining jiras and push out those that don't have
much progress today.

Thanks all!
Weichiu


On Tue, Apr 20, 2021 at 12:57 PM Wei-Chiu Chuang <weichiu@cloudera.com>
wrote:

> Billie and Viraj,
>
> As of today we are at 657 resolved jiras in 3.3.1
> https://issues.apache.org/jira/issues/?filter=-1&jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20status%20in%20(Resolved)%20AND%20fixVersion%20in%20(3.3.1)
> There are so many new features (
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.3+Release) I
> counted 10, that it is barely a dot release.
>
> Please make every effort to backport the said jiras to branch-3.3. I'd
> like to cut the branch sooner than later. I'd also call for more frequent
> releases. At the current development pace, a dot release every quarter
> would make sense IMO.
>
> Given the number of release blockers, I propose a tentative code freeze
> date: end of next Friday. After that I'd like to cut the branch and make an
> RC.
>
> Thoughts? I can't do this all alone. If you find things that are worth
> stopping the release, please tag the jira with the label 'release-blocker'.
>
>
>
> On Tue, Apr 20, 2021 at 2:44 AM Billie Rinaldi <billie@apache.org> wrote:
>
>> I was thinking of backporting HADOOP-16948 for 3.3.1.
>>
>> Billie
>>
>> On Mon, Apr 19, 2021 at 1:33 AM Wei-Chiu Chuang
>> <weichiu@cloudera.com.invalid> wrote:
>>
>> > Hello, reviving this thread.
>> >
>> > I created a dashboard for Hadoop 3.3.1 release.
>> >
>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336122
>> > Also a jira to track the release work: HADOOP-17647
>> > <https://issues.apache.org/jira/browse/HADOOP-17647>
>> >
>> > We are currently at 5 release blockers and 3 critical issues for Hadoop
>> > 3.3.1. I'll go through each of them and push out the ones that aren't
>> > really blocking us.
>> >
>> > If you believe there are more features/bug fixes we should include in
>> 3.3.1
>> > (I spent the past few weeks backporting jiras but I'm sure I missed
>> some)
>> > please shout out.
>> >
>> > Meanwhile, I believe we need to release hadoop-thirdparty 1.1.0 too.
>> There
>> > are a number of tasks to be done there too. Let's start another thread
>> for
>> > hadoop-thirdparty 1.1.0 release.
>> >
>> > On Mon, Mar 15, 2021 at 7:04 PM hemanth boyina <
>> hemanthboyina224@gmail.com
>> > >
>> > wrote:
>> >
>> > > Hi Steve and Wei-Chiu
>> > >
>> > > Regarding the IPV6,Few years back we have rebased the HADOOP-11890 to
>> > trunk
>> > > and tried to work out with IPV6, we have faced some issues and have
>> made
>> > > the required changes for ipv6 to work.After the changes were made we
>> have
>> > > tested the IPV6 changes on top of Ipv4 and Ipv6 machines and tested
>> > > rigorously.Its been quite a some time these changes were deployed in
>> > > production cluster and have been in use for extensive purpose.
>> > >
>> > > I think it's good time to add this feature.
>> > >
>> > > Thanks
>> > > Hemanth Boyina
>> > >
>> > >
>> > >
>> > > On Thu, 11 Mar 2021, 10:22 Vinayakumar B, <vinayakumarb@apache.org>
>> > wrote:
>> > >
>> > > > Hi David,
>> > > >
>> > > > >> Still hoping for help here:
>> > > >
>> > > > >> https://issues.apache.org/jira/browse/HDFS-15790
>> > > >
>> > > > I will raise a PR for the said solution soon (in a day or two).
>> > > >
>> > > > -Vinay
>> > > >
>> > > > On Thu, 11 Mar 2021 at 5:39 AM, David <dam6923@gmail.com> wrote:
>> > > >
>> > > > > Hello,
>> > > > >
>> > > > > Still hoping for help here:
>> > > > >
>> > > > > https://issues.apache.org/jira/browse/HDFS-15790
>> > > > >
>> > > > > Looks like it has been worked on, not sure how to best move it
>> > forward.
>> > > > >
>> > > > > On Wed, Mar 10, 2021, 12:21 PM Steve Loughran
>> > > > <stevel@cloudera.com.invalid
>> > > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > I'm going to argue its too late to do IPv6 support close
to a
>> > > release,
>> > > > as
>> > > > > > it's best if its on developer machines for some time to
let all
>> > > quirks
>> > > > > > surface. It's not so much IPv6 itself, but do we cause any
>> > > regressions
>> > > > on
>> > > > > > IPv4?
>> > > > > >
>> > > > > > But: it can/should go into trunk and stabilize there
>> > > > > >
>> > > > > > On Thu, 4 Mar 2021 at 03:52, Muralikrishna Dmmkr <
>> > > > > > muralikrishna.dmmkr@gmail.com> wrote:
>> > > > > >
>> > > > > > > Hi Brahma,
>> > > > > > >
>> > > > > > > I have missed out mentioning about the IPV6 feature
in the
>> last
>> > > mail,
>> > > > > > > Support for IPV6 has been in development since 2015
and We
>> have
>> > > done
>> > > > a
>> > > > > > good
>> > > > > > > amount of testing at our organisation, the feature
is stable
>> and
>> > > used
>> > > > > by
>> > > > > > > our customers extensively in the last one year. I think
it is
>> a
>> > > good
>> > > > > time
>> > > > > > > to add the IPV6 support to 3.3.1.
>> > > > > > >
>> > > > > > > https://issues.apache.org/jira/browse/HADOOP-11890
>> > > > > > >
>> > > > > > > Thanks
>> > > > > > > D M Murali Krishna Reddy
>> > > > > > >
>> > > > > > > On Wed, Feb 24, 2021 at 9:13 AM Muralikrishna Dmmkr
<
>> > > > > > > muralikrishna.dmmkr@gmail.com> wrote:
>> > > > > > >
>> > > > > > > > Hi Brahma,
>> > > > > > > >
>> > > > > > > > Can we have this new feature "YARN Registry based
AM
>> discovery
>> > > with
>> > > > > > retry
>> > > > > > > > and in-flight task persistent via JHS" in the
upcoming 3.1.1
>> > > > > release. I
>> > > > > > > > have also attached a test-report in the below
jira.
>> > > > > > > >
>> > > > > > > > https://issues.apache.org/jira/browse/MAPREDUCE-6726
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > D M Murali Krishna Reddy
>> > > > > > > >
>> > > > > > > > On Tue, Feb 23, 2021 at 10:11 AM Brahma Reddy
Battula <
>> > > > > > brahma@apache.org
>> > > > > > > >
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > >> Hi Bilwa,
>> > > > > > > >>
>> > > > > > > >> I have commented on the jira's you mentioned.
Based on the
>> > > > stability
>> > > > > > we
>> > > > > > > >> can
>> > > > > > > >> plan this.But needs to be merged ASAP.
>> > > > > > > >>
>> > > > > > > >>
>> > > > > > > >>
>> > > > > > > >> On Fri, Feb 19, 2021 at 5:20 PM bilwa st <
>> stbilwa@gmail.com>
>> > > > wrote:
>> > > > > > > >>
>> > > > > > > >> > Hi Brahma,
>> > > > > > > >> >
>> > > > > > > >> > Can we have below features in 3.3.1 release?
We have been
>> > > using
>> > > > > > these
>> > > > > > > >> > features for a long time. They are stable
and tested in
>> > bigger
>> > > > > > > clusters.
>> > > > > > > >> >
>> > > > > > > >> > 1. Container reuse -
>> > > > > > > >> https://issues.apache.org/jira/browse/MAPREDUCE-6749
>> > > > > > > >> > 2. Speculative attempts should not run
on the same node -
>> > > > > > > >> > https://issues.apache.org/jira/browse/MAPREDUCE-7169
>> > > > > > > >> >
>> > > > > > > >> > Thanks,
>> > > > > > > >> > Bilwa
>> > > > > > > >> >
>> > > > > > > >> > On Thu, Feb 18, 2021, 1:49 PM Brahma
Reddy Battula <
>> > > > > > brahma@apache.org
>> > > > > > > >
>> > > > > > > >> > wrote:
>> > > > > > > >> >
>> > > > > > > >> >> Sorry for the late reply..
>> > > > > > > >> >>
>> > > > > > > >> >> I will come up with a plan.. Please
let me know if
>> anybody
>> > > has
>> > > > > some
>> > > > > > > >> >> features/improvements/bugs that need
to be included.
>> > > > > > > >> >>
>> > > > > > > >> >> On Mon, Feb 15, 2021 at 9:39 PM Sunil
Govindan <
>> > > > > sunilg@apache.org>
>> > > > > > > >> wrote:
>> > > > > > > >> >>
>> > > > > > > >> >> > Hi Wei-Chiu,
>> > > > > > > >> >> >
>> > > > > > > >> >> > What will be the next steps
here for 3.3.1 planning?
>> > > > > > > >> >> >
>> > > > > > > >> >> > Thanks
>> > > > > > > >> >> > Sunil
>> > > > > > > >> >> >
>> > > > > > > >> >> > On Mon, Feb 8, 2021 at 11:56
PM Stack <
>> stack@duboce.net>
>> > > > > wrote:
>> > > > > > > >> >> >
>> > > > > > > >> >> > > On Wed, Feb 3, 2021 at
6:41 AM Steve Loughran
>> > > > > > > >> >> > <stevel@cloudera.com.invalid
>> > > > > > > >> >> > > >
>> > > > > > > >> >> > > wrote:
>> > > > > > > >> >> > >
>> > > > > > > >> >> > > >
>> > > > > > > >> >> > > > Regarding blockers
&c: how about we have a little
>> > > > hackathon
>> > > > > > > >> where we
>> > > > > > > >> >> > try
>> > > > > > > >> >> > > > and get things in.
This means a promise of review
>> > time
>> > > > from
>> > > > > > the
>> > > > > > > >> >> people
>> > > > > > > >> >> > > with
>> > > > > > > >> >> > > > commit rights and
other people who understand the
>> > code
>> > > > > > (Stack?)
>> > > > > > > >> >> > > >
>> > > > > > > >> >> > > >
>> > > > > > > >> >> > >
>> > > > > > > >> >> > > I'm up for helping get
3.3.1 out (reviewing,
>> hackathon,
>> > > > > > testing).
>> > > > > > > >> >> > > Thanks,
>> > > > > > > >> >> > > S
>> > > > > > > >> >> > >
>> > > > > > > >> >> > >
>> > > > > > > >> >> > >
>> > > > > > > >> >> > >
>> > > > > > > >> >> > > > -steve
>> > > > > > > >> >> > > >
>> > > > > > > >> >> > > > On Thu, 28 Jan 2021
at 06:48, Ayush Saxena <
>> > > > > > ayushtkn@gmail.com
>> > > > > > > >
>> > > > > > > >> >> wrote:
>> > > > > > > >> >> > > >
>> > > > > > > >> >> > > > > +1
>> > > > > > > >> >> > > > > Just to mention
we would need to release
>> > > > > hadoop-thirdparty
>> > > > > > > too
>> > > > > > > >> >> > before.
>> > > > > > > >> >> > > > > Presently we
are using the snapshot version of
>> it.
>> > > > > > > >> >> > > > >
>> > > > > > > >> >> > > > > -Ayush
>> > > > > > > >> >> > > > >
>> > > > > > > >> >> > > > > > On 28-Jan-2021,
at 6:59 AM, Wei-Chiu Chuang <
>> > > > > > > >> weichiu@apache.org
>> > > > > > > >> >> >
>> > > > > > > >> >> > > > wrote:
>> > > > > > > >> >> > > > > >
>> > > > > > > >> >> > > > > > ´╗┐Hi all,
>> > > > > > > >> >> > > > > >
>> > > > > > > >> >> > > > > > Hadoop 3.3.0
was released half a year ago,
>> and as
>> > > of
>> > > > > now
>> > > > > > > >> we've
>> > > > > > > >> >> > > > > accumulated
>> > > > > > > >> >> > > > > > more than
400 changes in the branch-3.3. A
>> number
>> > > of
>> > > > > > > >> >> downstreamers
>> > > > > > > >> >> > > are
>> > > > > > > >> >> > > > > > eagerly
waiting for 3.3.1 which addresses the
>> > guava
>> > > > > > version
>> > > > > > > >> >> > conflict
>> > > > > > > >> >> > > > > issue.
>> > > > > > > >> >> > > > > >
>> > > > > > > >> >> > > > > >
>> > > > > > > >> >> > > > >
>> > > > > > > >> >> > > >
>> > > > > > > >> >> > >
>> > > > > > > >> >> >
>> > > > > > > >> >>
>> > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/issues/?filter=-1&jql=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20and%20fixVersion%20in%20(3.3.1)%20and%20status%20%3D%20Resolved%20
>> > > > > > > >> >> > > > > >
>> > > > > > > >> >> > > > > > We should
start the release work for 3.3.1
>> before
>> > > the
>> > > > > > diff
>> > > > > > > >> >> becomes
>> > > > > > > >> >> > > even
>> > > > > > > >> >> > > > > > larger.
>> > > > > > > >> >> > > > > >
>> > > > > > > >> >> > > > > > I believe
there are  currently only two real
>> > > blockers
>> > > > > > for a
>> > > > > > > >> >> 3.3.1
>> > > > > > > >> >> > > > (using
>> > > > > > > >> >> > > > > > this filter
>> > > > > > > >> >> > > > > >
>> > > > > > > >> >> > > > >
>> > > > > > > >> >> > > >
>> > > > > > > >> >> > >
>> > > > > > > >> >> >
>> > > > > > > >> >>
>> > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/issues/?filter=-1&jql=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20AND%20cf%5B12310320%5D%20in%20(3.3.1)%20AND%20status%20not%20in%20(Resolved)%20ORDER%20BY%20priority%20DESC
>> > > > > > > >> >> > > > > > )
>> > > > > > > >> >> > > > > >
>> > > > > > > >> >> > > > > >
>> > > > > > > >> >> > > > > >   1. HDFS-15566
<
>> > > > > > > >> >> https://issues.apache.org/jira/browse/HDFS-15566>
>> > > > > > > >> >> > > > > >   2.
>> > > > > > > >> >> > > > > >      1.
HADOOP-17112 <
>> > > > > > > >> >> > > >
>> https://issues.apache.org/jira/browse/HADOOP-17112
>> > > > > > > >> >> > > > > >
>> > > > > > > >> >> > > > > >        
2.
>> > > > > > > >> >> > > > > >
>> > > > > > > >> >> > > > > >
>> > > > > > > >> >> > > > > >
>> > > > > > > >> >> > > > > > Is there
anyone who would volunteer to be the
>> > 3.3.1
>> > > > RM?
>> > > > > > > >> >> > > > > >
>> > > > > > > >> >> > > > > > Also, the
HowToRelease wiki does not describe
>> the
>> > > ARM
>> > > > > > build
>> > > > > > > >> >> > process.
>> > > > > > > >> >> > > > > That's
>> > > > > > > >> >> > > > > > going to
be important for future releases.
>> > > > > > > >> >> > > > >
>> > > > > > > >> >> > > > >
>> > > > > > > >> >>
>> > > > > >
>> > ---------------------------------------------------------------------
>> > > > > > > >> >> > > > > To unsubscribe,
e-mail:
>> > > > > > > hdfs-dev-unsubscribe@hadoop.apache.org
>> > > > > > > >> >> > > > > For additional
commands, e-mail:
>> > > > > > > >> hdfs-dev-help@hadoop.apache.org
>> > > > > > > >> >> > > > >
>> > > > > > > >> >> > > > >
>> > > > > > > >> >> > > >
>> > > > > > > >> >> > >
>> > > > > > > >> >> >
>> > > > > > > >> >>
>> > > > > > > >> >>
>> > > > > > > >> >> --
>> > > > > > > >> >>
>> > > > > > > >> >>
>> > > > > > > >> >>
>> > > > > > > >> >> --Brahma Reddy Battula
>> > > > > > > >> >>
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > > >> --
>> > > > > > > >>
>> > > > > > > >>
>> > > > > > > >>
>> > > > > > > >> --Brahma Reddy Battula
>> > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > --
>> > > > -Vinay
>> > > >
>> > >
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message