hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wei-Chiu Chuang <weic...@cloudera.com.INVALID>
Subject Re: [DISCUSS] Hadoop 3.3.1 release
Date Wed, 19 May 2021 04:24:38 GMT
Back at this. The Hadoop thirdparty 1.1.0 is released.

We have one real blocker for the 3.3.1 release (HDFS-15790 Make
ProtobufRpcEngineProtos and ProtobufRpcEngineProtos2 Co-Exist
<https://issues.apache.org/jira/browse/HDFS-15790>)
I tried to rebase the PR but it didn't work like I wanted. Some help is
needed there. If either Vinay comes back to rebase the PR, or if someone
else can take over the PR, that would be great.

Meanwhile, I'll cut a branch for 3.3.1 later. If everything works out fine,
I should be able to make a RC0 by tonight (which doesn't contain
HDFS-15790) so you guys can quickly check out.

The first RC will probably not have the aarch64 binary. I don't have much
experience with that.

Cheers,
Wei-Chiu



On Thu, May 13, 2021 at 9:04 AM Wei-Chiu Chuang <weichiu@cloudera.com>
wrote:

> Hello it's me again.
>
> While working on the hadoop-thirdparty release, I would like to make
> progress on the Hadoop 3.3.1 release in parallel as well.
> Looking at the jira dashboard, I think the only real blocker is HDFS-15790
> https://github.com/apache/hadoop/pull/2767
>
> I'll try to review the remaining jiras and push out those that don't have
> much progress today.
>
> Thanks all!
> Weichiu
>
>
> On Tue, Apr 20, 2021 at 12:57 PM Wei-Chiu Chuang <weichiu@cloudera.com>
> wrote:
>
>> Billie and Viraj,
>>
>> As of today we are at 657 resolved jiras in 3.3.1
>> https://issues.apache.org/jira/issues/?filter=-1&jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20status%20in%20(Resolved)%20AND%20fixVersion%20in%20(3.3.1)
>> There are so many new features (
>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.3+Release) I
>> counted 10, that it is barely a dot release.
>>
>> Please make every effort to backport the said jiras to branch-3.3. I'd
>> like to cut the branch sooner than later. I'd also call for more frequent
>> releases. At the current development pace, a dot release every quarter
>> would make sense IMO.
>>
>> Given the number of release blockers, I propose a tentative code freeze
>> date: end of next Friday. After that I'd like to cut the branch and make an
>> RC.
>>
>> Thoughts? I can't do this all alone. If you find things that are worth
>> stopping the release, please tag the jira with the label 'release-blocker'.
>>
>>
>>
>> On Tue, Apr 20, 2021 at 2:44 AM Billie Rinaldi <billie@apache.org> wrote:
>>
>>> I was thinking of backporting HADOOP-16948 for 3.3.1.
>>>
>>> Billie
>>>
>>> On Mon, Apr 19, 2021 at 1:33 AM Wei-Chiu Chuang
>>> <weichiu@cloudera.com.invalid> wrote:
>>>
>>> > Hello, reviving this thread.
>>> >
>>> > I created a dashboard for Hadoop 3.3.1 release.
>>> >
>>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336122
>>> > Also a jira to track the release work: HADOOP-17647
>>> > <https://issues.apache.org/jira/browse/HADOOP-17647>
>>> >
>>> > We are currently at 5 release blockers and 3 critical issues for Hadoop
>>> > 3.3.1. I'll go through each of them and push out the ones that aren't
>>> > really blocking us.
>>> >
>>> > If you believe there are more features/bug fixes we should include in
>>> 3.3.1
>>> > (I spent the past few weeks backporting jiras but I'm sure I missed
>>> some)
>>> > please shout out.
>>> >
>>> > Meanwhile, I believe we need to release hadoop-thirdparty 1.1.0 too.
>>> There
>>> > are a number of tasks to be done there too. Let's start another thread
>>> for
>>> > hadoop-thirdparty 1.1.0 release.
>>> >
>>> > On Mon, Mar 15, 2021 at 7:04 PM hemanth boyina <
>>> hemanthboyina224@gmail.com
>>> > >
>>> > wrote:
>>> >
>>> > > Hi Steve and Wei-Chiu
>>> > >
>>> > > Regarding the IPV6,Few years back we have rebased the HADOOP-11890
to
>>> > trunk
>>> > > and tried to work out with IPV6, we have faced some issues and have
>>> made
>>> > > the required changes for ipv6 to work.After the changes were made we
>>> have
>>> > > tested the IPV6 changes on top of Ipv4 and Ipv6 machines and tested
>>> > > rigorously.Its been quite a some time these changes were deployed in
>>> > > production cluster and have been in use for extensive purpose.
>>> > >
>>> > > I think it's good time to add this feature.
>>> > >
>>> > > Thanks
>>> > > Hemanth Boyina
>>> > >
>>> > >
>>> > >
>>> > > On Thu, 11 Mar 2021, 10:22 Vinayakumar B, <vinayakumarb@apache.org>
>>> > wrote:
>>> > >
>>> > > > Hi David,
>>> > > >
>>> > > > >> Still hoping for help here:
>>> > > >
>>> > > > >> https://issues.apache.org/jira/browse/HDFS-15790
>>> > > >
>>> > > > I will raise a PR for the said solution soon (in a day or two).
>>> > > >
>>> > > > -Vinay
>>> > > >
>>> > > > On Thu, 11 Mar 2021 at 5:39 AM, David <dam6923@gmail.com>
wrote:
>>> > > >
>>> > > > > Hello,
>>> > > > >
>>> > > > > Still hoping for help here:
>>> > > > >
>>> > > > > https://issues.apache.org/jira/browse/HDFS-15790
>>> > > > >
>>> > > > > Looks like it has been worked on, not sure how to best move
it
>>> > forward.
>>> > > > >
>>> > > > > On Wed, Mar 10, 2021, 12:21 PM Steve Loughran
>>> > > > <stevel@cloudera.com.invalid
>>> > > > > >
>>> > > > > wrote:
>>> > > > >
>>> > > > > > I'm going to argue its too late to do IPv6 support close
to a
>>> > > release,
>>> > > > as
>>> > > > > > it's best if its on developer machines for some time
to let all
>>> > > quirks
>>> > > > > > surface. It's not so much IPv6 itself, but do we cause
any
>>> > > regressions
>>> > > > on
>>> > > > > > IPv4?
>>> > > > > >
>>> > > > > > But: it can/should go into trunk and stabilize there
>>> > > > > >
>>> > > > > > On Thu, 4 Mar 2021 at 03:52, Muralikrishna Dmmkr <
>>> > > > > > muralikrishna.dmmkr@gmail.com> wrote:
>>> > > > > >
>>> > > > > > > Hi Brahma,
>>> > > > > > >
>>> > > > > > > I have missed out mentioning about the IPV6 feature
in the
>>> last
>>> > > mail,
>>> > > > > > > Support for IPV6 has been in development since
2015 and We
>>> have
>>> > > done
>>> > > > a
>>> > > > > > good
>>> > > > > > > amount of testing at our organisation, the feature
is stable
>>> and
>>> > > used
>>> > > > > by
>>> > > > > > > our customers extensively in the last one year.
I think it
>>> is a
>>> > > good
>>> > > > > time
>>> > > > > > > to add the IPV6 support to 3.3.1.
>>> > > > > > >
>>> > > > > > > https://issues.apache.org/jira/browse/HADOOP-11890
>>> > > > > > >
>>> > > > > > > Thanks
>>> > > > > > > D M Murali Krishna Reddy
>>> > > > > > >
>>> > > > > > > On Wed, Feb 24, 2021 at 9:13 AM Muralikrishna Dmmkr
<
>>> > > > > > > muralikrishna.dmmkr@gmail.com> wrote:
>>> > > > > > >
>>> > > > > > > > Hi Brahma,
>>> > > > > > > >
>>> > > > > > > > Can we have this new feature "YARN Registry
based AM
>>> discovery
>>> > > with
>>> > > > > > retry
>>> > > > > > > > and in-flight task persistent via JHS" in
the upcoming
>>> 3.1.1
>>> > > > > release. I
>>> > > > > > > > have also attached a test-report in the below
jira.
>>> > > > > > > >
>>> > > > > > > > https://issues.apache.org/jira/browse/MAPREDUCE-6726
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > Thanks,
>>> > > > > > > > D M Murali Krishna Reddy
>>> > > > > > > >
>>> > > > > > > > On Tue, Feb 23, 2021 at 10:11 AM Brahma Reddy
Battula <
>>> > > > > > brahma@apache.org
>>> > > > > > > >
>>> > > > > > > > wrote:
>>> > > > > > > >
>>> > > > > > > >> Hi Bilwa,
>>> > > > > > > >>
>>> > > > > > > >> I have commented on the jira's you mentioned.
Based on the
>>> > > > stability
>>> > > > > > we
>>> > > > > > > >> can
>>> > > > > > > >> plan this.But needs to be merged ASAP.
>>> > > > > > > >>
>>> > > > > > > >>
>>> > > > > > > >>
>>> > > > > > > >> On Fri, Feb 19, 2021 at 5:20 PM bilwa
st <
>>> stbilwa@gmail.com>
>>> > > > wrote:
>>> > > > > > > >>
>>> > > > > > > >> > Hi Brahma,
>>> > > > > > > >> >
>>> > > > > > > >> > Can we have below features in 3.3.1
release? We have
>>> been
>>> > > using
>>> > > > > > these
>>> > > > > > > >> > features for a long time. They are
stable and tested in
>>> > bigger
>>> > > > > > > clusters.
>>> > > > > > > >> >
>>> > > > > > > >> > 1. Container reuse -
>>> > > > > > > >> https://issues.apache.org/jira/browse/MAPREDUCE-6749
>>> > > > > > > >> > 2. Speculative attempts should not
run on the same node
>>> -
>>> > > > > > > >> > https://issues.apache.org/jira/browse/MAPREDUCE-7169
>>> > > > > > > >> >
>>> > > > > > > >> > Thanks,
>>> > > > > > > >> > Bilwa
>>> > > > > > > >> >
>>> > > > > > > >> > On Thu, Feb 18, 2021, 1:49 PM Brahma
Reddy Battula <
>>> > > > > > brahma@apache.org
>>> > > > > > > >
>>> > > > > > > >> > wrote:
>>> > > > > > > >> >
>>> > > > > > > >> >> Sorry for the late reply..
>>> > > > > > > >> >>
>>> > > > > > > >> >> I will come up with a plan..
Please let me know if
>>> anybody
>>> > > has
>>> > > > > some
>>> > > > > > > >> >> features/improvements/bugs that
need to be included.
>>> > > > > > > >> >>
>>> > > > > > > >> >> On Mon, Feb 15, 2021 at 9:39
PM Sunil Govindan <
>>> > > > > sunilg@apache.org>
>>> > > > > > > >> wrote:
>>> > > > > > > >> >>
>>> > > > > > > >> >> > Hi Wei-Chiu,
>>> > > > > > > >> >> >
>>> > > > > > > >> >> > What will be the next steps
here for 3.3.1 planning?
>>> > > > > > > >> >> >
>>> > > > > > > >> >> > Thanks
>>> > > > > > > >> >> > Sunil
>>> > > > > > > >> >> >
>>> > > > > > > >> >> > On Mon, Feb 8, 2021 at 11:56
PM Stack <
>>> stack@duboce.net>
>>> > > > > wrote:
>>> > > > > > > >> >> >
>>> > > > > > > >> >> > > On Wed, Feb 3, 2021
at 6:41 AM Steve Loughran
>>> > > > > > > >> >> > <stevel@cloudera.com.invalid
>>> > > > > > > >> >> > > >
>>> > > > > > > >> >> > > wrote:
>>> > > > > > > >> >> > >
>>> > > > > > > >> >> > > >
>>> > > > > > > >> >> > > > Regarding blockers
&c: how about we have a little
>>> > > > hackathon
>>> > > > > > > >> where we
>>> > > > > > > >> >> > try
>>> > > > > > > >> >> > > > and get things
in. This means a promise of review
>>> > time
>>> > > > from
>>> > > > > > the
>>> > > > > > > >> >> people
>>> > > > > > > >> >> > > with
>>> > > > > > > >> >> > > > commit rights
and other people who understand the
>>> > code
>>> > > > > > (Stack?)
>>> > > > > > > >> >> > > >
>>> > > > > > > >> >> > > >
>>> > > > > > > >> >> > >
>>> > > > > > > >> >> > > I'm up for helping
get 3.3.1 out (reviewing,
>>> hackathon,
>>> > > > > > testing).
>>> > > > > > > >> >> > > Thanks,
>>> > > > > > > >> >> > > S
>>> > > > > > > >> >> > >
>>> > > > > > > >> >> > >
>>> > > > > > > >> >> > >
>>> > > > > > > >> >> > >
>>> > > > > > > >> >> > > > -steve
>>> > > > > > > >> >> > > >
>>> > > > > > > >> >> > > > On Thu, 28 Jan
2021 at 06:48, Ayush Saxena <
>>> > > > > > ayushtkn@gmail.com
>>> > > > > > > >
>>> > > > > > > >> >> wrote:
>>> > > > > > > >> >> > > >
>>> > > > > > > >> >> > > > > +1
>>> > > > > > > >> >> > > > > Just to mention
we would need to release
>>> > > > > hadoop-thirdparty
>>> > > > > > > too
>>> > > > > > > >> >> > before.
>>> > > > > > > >> >> > > > > Presently
we are using the snapshot version of
>>> it.
>>> > > > > > > >> >> > > > >
>>> > > > > > > >> >> > > > > -Ayush
>>> > > > > > > >> >> > > > >
>>> > > > > > > >> >> > > > > > On 28-Jan-2021,
at 6:59 AM, Wei-Chiu Chuang <
>>> > > > > > > >> weichiu@apache.org
>>> > > > > > > >> >> >
>>> > > > > > > >> >> > > > wrote:
>>> > > > > > > >> >> > > > > >
>>> > > > > > > >> >> > > > > > ´╗┐Hi
all,
>>> > > > > > > >> >> > > > > >
>>> > > > > > > >> >> > > > > > Hadoop
3.3.0 was released half a year ago,
>>> and as
>>> > > of
>>> > > > > now
>>> > > > > > > >> we've
>>> > > > > > > >> >> > > > > accumulated
>>> > > > > > > >> >> > > > > > more
than 400 changes in the branch-3.3. A
>>> number
>>> > > of
>>> > > > > > > >> >> downstreamers
>>> > > > > > > >> >> > > are
>>> > > > > > > >> >> > > > > > eagerly
waiting for 3.3.1 which addresses the
>>> > guava
>>> > > > > > version
>>> > > > > > > >> >> > conflict
>>> > > > > > > >> >> > > > > issue.
>>> > > > > > > >> >> > > > > >
>>> > > > > > > >> >> > > > > >
>>> > > > > > > >> >> > > > >
>>> > > > > > > >> >> > > >
>>> > > > > > > >> >> > >
>>> > > > > > > >> >> >
>>> > > > > > > >> >>
>>> > > > > > > >>
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://issues.apache.org/jira/issues/?filter=-1&jql=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20and%20fixVersion%20in%20(3.3.1)%20and%20status%20%3D%20Resolved%20
>>> > > > > > > >> >> > > > > >
>>> > > > > > > >> >> > > > > > We should
start the release work for 3.3.1
>>> before
>>> > > the
>>> > > > > > diff
>>> > > > > > > >> >> becomes
>>> > > > > > > >> >> > > even
>>> > > > > > > >> >> > > > > > larger.
>>> > > > > > > >> >> > > > > >
>>> > > > > > > >> >> > > > > > I believe
there are  currently only two real
>>> > > blockers
>>> > > > > > for a
>>> > > > > > > >> >> 3.3.1
>>> > > > > > > >> >> > > > (using
>>> > > > > > > >> >> > > > > > this
filter
>>> > > > > > > >> >> > > > > >
>>> > > > > > > >> >> > > > >
>>> > > > > > > >> >> > > >
>>> > > > > > > >> >> > >
>>> > > > > > > >> >> >
>>> > > > > > > >> >>
>>> > > > > > > >>
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://issues.apache.org/jira/issues/?filter=-1&jql=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20AND%20cf%5B12310320%5D%20in%20(3.3.1)%20AND%20status%20not%20in%20(Resolved)%20ORDER%20BY%20priority%20DESC
>>> > > > > > > >> >> > > > > > )
>>> > > > > > > >> >> > > > > >
>>> > > > > > > >> >> > > > > >
>>> > > > > > > >> >> > > > > >   1.
HDFS-15566 <
>>> > > > > > > >> >> https://issues.apache.org/jira/browse/HDFS-15566>
>>> > > > > > > >> >> > > > > >   2.
>>> > > > > > > >> >> > > > > >    
 1. HADOOP-17112 <
>>> > > > > > > >> >> > > >
>>> https://issues.apache.org/jira/browse/HADOOP-17112
>>> > > > > > > >> >> > > > > >
>>> > > > > > > >> >> > > > > >    
    2.
>>> > > > > > > >> >> > > > > >
>>> > > > > > > >> >> > > > > >
>>> > > > > > > >> >> > > > > >
>>> > > > > > > >> >> > > > > > Is there
anyone who would volunteer to be the
>>> > 3.3.1
>>> > > > RM?
>>> > > > > > > >> >> > > > > >
>>> > > > > > > >> >> > > > > > Also,
the HowToRelease wiki does not
>>> describe the
>>> > > ARM
>>> > > > > > build
>>> > > > > > > >> >> > process.
>>> > > > > > > >> >> > > > > That's
>>> > > > > > > >> >> > > > > > going
to be important for future releases.
>>> > > > > > > >> >> > > > >
>>> > > > > > > >> >> > > > >
>>> > > > > > > >> >>
>>> > > > > >
>>> > ---------------------------------------------------------------------
>>> > > > > > > >> >> > > > > To unsubscribe,
e-mail:
>>> > > > > > > hdfs-dev-unsubscribe@hadoop.apache.org
>>> > > > > > > >> >> > > > > For additional
commands, e-mail:
>>> > > > > > > >> hdfs-dev-help@hadoop.apache.org
>>> > > > > > > >> >> > > > >
>>> > > > > > > >> >> > > > >
>>> > > > > > > >> >> > > >
>>> > > > > > > >> >> > >
>>> > > > > > > >> >> >
>>> > > > > > > >> >>
>>> > > > > > > >> >>
>>> > > > > > > >> >> --
>>> > > > > > > >> >>
>>> > > > > > > >> >>
>>> > > > > > > >> >>
>>> > > > > > > >> >> --Brahma Reddy Battula
>>> > > > > > > >> >>
>>> > > > > > > >> >
>>> > > > > > > >>
>>> > > > > > > >> --
>>> > > > > > > >>
>>> > > > > > > >>
>>> > > > > > > >>
>>> > > > > > > >> --Brahma Reddy Battula
>>> > > > > > > >>
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > --
>>> > > > -Vinay
>>> > > >
>>> > >
>>> >
>>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message