hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wangda Tan <wheele...@gmail.com>
Subject Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Date Wed, 04 Apr 2018 03:13:40 GMT
Hi Vinod / Arpit,

I checked following versions:
- 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:

Jars in maven repo [1] are *always* different from jars in the binary
tarball [2]: (I only checked hadoop-yarn-api-version.jar)

(Following numbers are sizes of the jar)
2.6.5:
- Jar in Maven: 1896185
- Jar in tarball: 1891485

2.7.5:
- Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
- Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
(Even size is same, md5 is different)

2.8.3:
- Jar in Maven: 2451433
- Jar in tarball: 2438975

2.9.0:
- Jar in Maven: 2791477
- Jar in tarball: 2777789

3.0.1:
- Jar in Maven: 2852604
- Jar in tarball: 2851373

I guess the differences come from our release process.

Thanks,
Wangda

[1] Maven jars are downloaded from
https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
<version>/hadoop-yarn-api-<version>.jar
[2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/


On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org>
wrote:

> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheeleast@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com>
> wrote:
>
>> Thanks Wangda, I see the shaded jars now.
>>
>> Are the repo jars required to be the same as the binary release? They
>> don’t match right now, probably they got rebuilt.
>>
>> +1 (binding), modulo that remaining question.
>>
>> * Verified signatures
>> * Verified checksums for source and binary artefacts
>> * Sanity checked jars on r.a.o.
>> * Built from source
>> * Deployed to 3 node secure cluster with NameNode HA
>> * Verified HDFS web UIs
>> * Tried out HDFS shell commands
>> * Ran sample MapReduce jobs
>>
>> Thanks!
>>
>>
>> ----------------------------------------------------------------------
>> From: Wangda Tan <wheeleast@gmail.com>
>> Date: Monday, April 2, 2018 at 9:25 PM
>> To: Arpit Agarwal <aagarwal@hortonworks.com>
>> Cc: Gera Shegalov <gera05@gmail.com>, Sunil G <sunilg@apache.org>, "
>> yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>, Hdfs-dev <
>> hdfs-dev@hadoop.apache.org>, Hadoop Common <common-dev@hadoop.apache.org>,
>> "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
>> Vinod Kumar Vavilapalli <vinodkv@apache.org>
>> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>
>> As pointed by Arpit, the previously deployed shared jars are incorrect.
>> Just redeployed jars and staged. @Arpit, could you please check the updated
>> Maven repo? https://repository.apache.org/content/repositories/
>> orgapachehadoop-1092
>>
>> Since the jars inside binary tarballs are correct (
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
>> need roll another RC, just update Maven repo should be sufficient.
>>
>> Best,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheeleast@gmail.com>
>> wrote:
>> Hi Arpit,
>>
>> Thanks for pointing out this.
>>
>> I just removed all .md5 files from artifacts. I found md5 checksums still
>> exist in .mds files and I didn't remove them from .mds file because it is
>> generated by create-release script and Apache guidance is "should not"
>> instead of "must not". Please let me know if you think they need to be
>> removed as well.
>>
>> - Wangda
>>
>>
>>
>> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
>> aagarwal@hortonworks.com> wrote:
>> Thanks for putting together this RC, Wangda.
>>
>> The guidance from Apache is to omit MD5s, specifically:
>>   > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>>
>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheeleast@gmail.com>
>> wrote:
>>
>> Hi Gera,
>>
>> It's my bad, I thought only src/bin tarball is enough.
>>
>> I just uploaded all other things under artifact/ to
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>
>> Please let me know if you have any other comments.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:gera05@gmail.com>
>> wrote:
>>
>>
>> Thanks, Wangda!
>>
>> There are many more artifacts in previous votes, e.g., see
>> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others the
>> site tarball is missing.
>>
>> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sunilg@apache.org> wrote:
>>
>>
>> Thanks Wangda for initiating the release.
>>
>> I tested this RC built from source file.
>>
>>
>>   - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>> UI.
>>   - Below feature sanity is done
>>      - Application priority
>>      - Application timeout
>>      - Intra Queue preemption with priority based
>>      - DS based affinity tests to verify placement constraints.
>>   - Tested basic NodeLabel scenarios.
>>      - Added couple of labels to few of nodes and behavior is coming
>>      correct.
>>      - Verified old UI  and new YARN UI for labels.
>>      - Submitted apps to labelled cluster and it works fine.
>>      - Also performed few cli commands related to nodelabel.
>>   - Test basic HA cases and seems correct.
>>   - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>> - Sunil
>>
>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheeleast@gmail.com>
>> wrote:
>>
>>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1].
>> We've
>>
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via http://repository.apache.org at
>> https://repository.apache.org/content/repositories/
>> orgapachehadoop-1090/
>>
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services,
>> Support
>>
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>>
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>> as
>>
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>> BY
>>
>> fixVersion ASC
>> [3]
>>
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message