hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Date Wed, 04 Apr 2018 18:48:15 GMT
that's "dangerously interesting". I think you are right, and I also think it'll just be the
version files which get generated

anyway, +1 binding

* ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer binding, then my
downstream tests. All is well, provided you also have a spark hive JAR patched to accept hadoop
3 as a legitimate hadoop version. That's an ongoing issue in the Spark project. With that
JAR on my CP my downstream tests were all happy (yesterday)

* today the staging files seem to be missing, at least maven is unable to find them even when
I turn the spark snapshots-and-staging profile on. That'll be the maven dist process at play,
nothing else

On 4 Apr 2018, at 04:13, Wangda Tan <wheeleast@gmail.com<mailto:wheeleast@gmail.com>>

Hi Vinod / Arpit,

I checked following versions:
- 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:

Jars in maven repo [1] are *always* different from jars in the binary
tarball [2]: (I only checked hadoop-yarn-api-version.jar)

(Following numbers are sizes of the jar)
- Jar in Maven: 1896185
- Jar in tarball: 1891485

- Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
- Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
(Even size is same, md5 is different)

- Jar in Maven: 2451433
- Jar in tarball: 2438975

- Jar in Maven: 2791477
- Jar in tarball: 2777789

- Jar in Maven: 2852604
- Jar in tarball: 2851373

I guess the differences come from our release process.


[1] Maven jars are downloaded from
[2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/

On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org<mailto:vinodkv@apache.org>>

We vote on the source code. The binaries are convenience artifacts.

This is what I would do - (a) Just replace both the maven jars as well as
the binaries to be consistent and correct. And then (b) Give a couple more
days for folks who tested on the binaries to reverify - I count one such
clear vote as of now.


On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheeleast@gmail.com<mailto:wheeleast@gmail.com>>

HI Arpit,

I think it won't match if we do rebuild. It should be fine as far as
they're signed, correct? I don't see any policy doesn't allow this.


On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com<mailto:aagarwal@hortonworks.com>>

Thanks Wangda, I see the shaded jars now.

Are the repo jars required to be the same as the binary release? They
don’t match right now, probably they got rebuilt.

+1 (binding), modulo that remaining question.

* Verified signatures
* Verified checksums for source and binary artefacts
* Sanity checked jars on r.a.o.
* Built from source
* Deployed to 3 node secure cluster with NameNode HA
* Verified HDFS web UIs
* Tried out HDFS shell commands
* Ran sample MapReduce jobs


From: Wangda Tan <wheeleast@gmail.com<mailto:wheeleast@gmail.com>>
Date: Monday, April 2, 2018 at 9:25 PM
To: Arpit Agarwal <aagarwal@hortonworks.com<mailto:aagarwal@hortonworks.com>>
Cc: Gera Shegalov <gera05@gmail.com<mailto:gera05@gmail.com>>, Sunil G <sunilg@apache.org<mailto:sunilg@apache.org>>,
yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>" <yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>>,
Hdfs-dev <
hdfs-dev@hadoop.apache.org<mailto:hdfs-dev@hadoop.apache.org>>, Hadoop Common <common-dev@hadoop.apache.org<mailto:common-dev@hadoop.apache.org>>,
"mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org>" <mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org>>,
Vinod Kumar Vavilapalli <vinodkv@apache.org<mailto:vinodkv@apache.org>>
Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

As pointed by Arpit, the previously deployed shared jars are incorrect.
Just redeployed jars and staged. @Arpit, could you please check the updated
Maven repo? https://repository.apache.org/content/repositories/

Since the jars inside binary tarballs are correct (
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
need roll another RC, just update Maven repo should be sufficient.


On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheeleast@gmail.com>
Hi Arpit,

Thanks for pointing out this.

I just removed all .md5 files from artifacts. I found md5 checksums still
exist in .mds files and I didn't remove them from .mds file because it is
generated by create-release script and Apache guidance is "should not"
instead of "must not". Please let me know if you think they need to be
removed as well.

- Wangda

On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
aagarwal@hortonworks.com<mailto:aagarwal@hortonworks.com>> wrote:
Thanks for putting together this RC, Wangda.

The guidance from Apache is to omit MD5s, specifically:
SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).


On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheeleast@gmail.com>

Hi Gera,

It's my bad, I thought only src/bin tarball is enough.

I just uploaded all other things under artifact/ to

Please let me know if you have any other comments.


On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:gera05@gmail.com>

Thanks, Wangda!

There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others the
site tarball is missing.

On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sunilg@apache.org> wrote:

Thanks Wangda for initiating the release.

I tested this RC built from source file.

 - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
 - Below feature sanity is done
    - Application priority
    - Application timeout
    - Intra Queue preemption with priority based
    - DS based affinity tests to verify placement constraints.
 - Tested basic NodeLabel scenarios.
    - Added couple of labels to few of nodes and behavior is coming
    - Verified old UI  and new YARN UI for labels.
    - Submitted apps to labelled cluster and it works fine.
    - Also performed few cli commands related to nodelabel.
 - Test basic HA cases and seems correct.
 - Tested new YARN UI . All pages are getting loaded correctly.

- Sunil

On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheeleast@gmail.com>

Hi folks,

Thanks to the many who helped with this release since Dec 2017 [1].

created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:


The RC tag in git is release-3.1.0-RC1. Last git commit SHA is

The maven artifacts are available via http://repository.apache.org<http://repository.apache.org/>

This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.

3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services,

rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.

For 3.1.0 RC0 vote discussion, please see [3].

We’d like to use this as a starting release for 3.1.x [1], depending on

it goes, get it stabilized and potentially use a 3.1.1 in several weeks

the stable release.

We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.

My +1 to start.




[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER

fixVersion ASC


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message