From yarn-dev-return-28851-apmail-hadoop-yarn-dev-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 21 06:00:13 2017 Return-Path: X-Original-To: apmail-hadoop-yarn-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 839B9108A2 for ; Tue, 21 Nov 2017 06:00:13 +0000 (UTC) Received: (qmail 21486 invoked by uid 500); 21 Nov 2017 06:00:06 -0000 Delivered-To: apmail-hadoop-yarn-dev-archive@hadoop.apache.org Received: (qmail 20122 invoked by uid 500); 21 Nov 2017 06:00:05 -0000 Mailing-List: contact yarn-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list yarn-dev@hadoop.apache.org Received: (qmail 19992 invoked by uid 99); 21 Nov 2017 06:00:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Nov 2017 06:00:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 515091806A9; Tue, 21 Nov 2017 06:00:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.23 X-Spam-Level: *** X-Spam-Status: No, score=3.23 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=2, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id aaOPqhyo6EOw; Tue, 21 Nov 2017 06:00:01 +0000 (UTC) Received: from mail-lf0-f50.google.com (mail-lf0-f50.google.com [209.85.215.50]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 9C2EB60DE4; Tue, 21 Nov 2017 06:00:01 +0000 (UTC) Received: by mail-lf0-f50.google.com with SMTP id f131so4695599lff.12; Mon, 20 Nov 2017 22:00:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=rjbaJ5DpVx5O5wjSJpAidTyC8bd5Of7oUkAYtWMLpj4=; b=OSiCkMpBtnwyHUXdcOUJuk3YvuHvywsI5StlF9v0zJjVkOO7i5lnfUagG1vsWzvQ7v J6NU0eAK7P6Nx8TixkAOWjrQCWTeieZ3TW17lzqcbiUUESUxYYVELRTcxrl/wF16Bn9N 96nCfs3WOxuW4TWQ8XoARFhjcDVnHEfpBuYCc9PpjBkBfk6Imyvs7VOURreKm+izs/pH P0Qr0/+jqTxzkGPyufsC2vqiNDyyKdqz3h/3AjOYwIICqVtRhK2Zn0kIbh7dSW5fo3hj xkp+PZwhV5tWnckr7Sv0msnhvblkEog4ENhk0rZLU5Xrfv1lCOW9p3x8SCZEPNOMp0JC Ul3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=rjbaJ5DpVx5O5wjSJpAidTyC8bd5Of7oUkAYtWMLpj4=; b=JhTTlmkhSzX1UDRdoMfZgFsXcbQbke8YeJMvig/oewED4ImlDwrTRgQU358QodFvfd jFPJ9OEDo4WS//qx1z78y/mwizFkleqGgclmsBHpYaKx+iajUx/C3WjDM6wz3IebCXHb q5bx8zKi/pMBupWJ9d/rJT2MeDTR4gwvqAoI27WhaDFGV+sFUphi9Sez8BWf6fYuCyqN MVqoPAtveNLbNu/JW2pB8yha+yHc+w9MUishthmhWbSwV7CEnNejgVImCeqxGWhXuAaM KcmLpQp8tN9A0CXS86ADaxS9tz8APSnFtZ4QcFEPbI1eYrkedK4EosdYqYk6PSfLokc6 D9Fg== X-Gm-Message-State: AJaThX6rSQSZ6pwVpxoKWU3yiJVZfCr0bwO3mNCXXlexOuFPSPyywMQ9 J6xhuv6uMQB7ef+AOQQzxeaivq2tTYYuSLnXwRQ= X-Google-Smtp-Source: AGs4zMYvGWf88eT1CjftJZEink65AOrnK/0w46SVV73sHdV3emv0JEWkXsh4wTbMi996qDAvjnv9blQg39MMKSOzs4Q= X-Received: by 10.25.159.196 with SMTP id i187mr3490010lfe.150.1511244001071; Mon, 20 Nov 2017 22:00:01 -0800 (PST) MIME-Version: 1.0 Sender: sjlee0@gmail.com Received: by 10.25.83.140 with HTTP; Mon, 20 Nov 2017 21:59:40 -0800 (PST) In-Reply-To: References: From: Sangjin Lee Date: Mon, 20 Nov 2017 21:59:40 -0800 X-Google-Sender-Auth: eZBrYB8rPTHMNazXtgpC11lR0WY Message-ID: Subject: Re: [VOTE] Release Apache Hadoop 3.0.0 RC0 To: Andrew Wang Cc: Vinod Kumar Vavilapalli , "common-dev@hadoop.apache.org" , "yarn-dev@hadoop.apache.org" , "mapreduce-dev@hadoop.apache.org" , "hdfs-dev@hadoop.apache.org" Content-Type: multipart/alternative; boundary="001a11411bbeda5947055e77e926" --001a11411bbeda5947055e77e926 Content-Type: text/plain; charset="UTF-8" On Mon, Nov 20, 2017 at 9:46 PM, Andrew Wang wrote: > Thanks for the spot Sangjin. I think this bug introduced in create-release > by HADOOP-14835. The multi-pass maven build generates these dummy client > jars during the site build since skipShade is specified. > > This might be enough to cancel the RC. Thoughts? > IMO yes. This was one of the key features mentioned in the 3.0 release notes. I appreciate your effort for the release Andrew! > > Best, > Andrew > > On Mon, Nov 20, 2017 at 7:51 PM, Sangjin Lee wrote: > >> I checked the client jars that are supposed to contain shaded >> dependencies, and they don't look quite right: >> >> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-api-3.0.0.jar >> -rw-r--r-- 0 andrew andrew 44531 Nov 14 11:53 >> hadoop-3.0.0/share/hadoop/client/hadoop-client-api-3.0.0.jar >> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-runtime-3.0.0.jar >> -rw-r--r-- 0 andrew andrew 45533 Nov 14 11:53 >> hadoop-3.0.0/share/hadoop/client/hadoop-client-runtime-3.0.0.jar >> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-minicluster-3.0. >> 0.jar >> -rw-r--r-- 0 andrew andrew 47015 Nov 14 11:53 >> hadoop-3.0.0/share/hadoop/client/hadoop-client-minicluster-3.0.0.jar >> >> When I look at what's inside those jar, they only seem to include >> pom-related files with no class files. Am I missing something? >> >> When I build from the source with -Pdist, I do get much bigger jars: >> total 113760 >> -rw-r--r-- 1 sangjinlee 120039211 17055399 Nov 20 17:17 >> hadoop-client-api-3.0.0.jar >> -rw-r--r-- 1 sangjinlee 120039211 20451447 Nov 20 17:19 >> hadoop-client-minicluster-3.0.0.jar >> -rw-r--r-- 1 sangjinlee 120039211 20730866 Nov 20 17:18 >> hadoop-client-runtime-3.0.0.jar >> >> Sangjin >> >> On Mon, Nov 20, 2017 at 5:52 PM, Sangjin Lee wrote: >> >>> >>> >>> On Mon, Nov 20, 2017 at 5:26 PM, Vinod Kumar Vavilapalli < >>> vinodkv@apache.org> wrote: >>> >>>> Thanks for all the push, Andrew! >>>> >>>> Looking at the RC. Went through my usual check-list. Here's my summary. >>>> Will cast my final vote after comparing and validating my findings with >>>> others. >>>> >>>> Verification >>>> >>>> - [Check] Successful recompilation from source tar-ball >>>> - [Check] Signature verification >>>> - [Check] Generating dist tarballs from source tar-ball >>>> - [Check] Testing >>>> -- Start NN, DN, RM, NM, JHS, Timeline Service >>>> -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, >>>> grep, pi >>>> -- Tested CLIs to print nodes, apps etc and also navigated UIs >>>> >>>> Issues found during testing >>>> >>>> Major >>>> - The previously supported way of being able to use different >>>> tar-balls for different sub-modules is completely broken - common and HDFS >>>> tar.gz are completely empty. >>>> - Cannot enable new UI in YARN because it is under a non-default >>>> compilation flag. It should be on by default. >>>> - One decommissioned node in YARN ResourceManager UI always appears to >>>> start with, even when there are no NodeManagers that are started yet: Info >>>> :-1, DECOMMISSIONED, null rack. It shows up only in the UI though, not >>>> in the CLI node -list >>>> >>>> Minor >>>> - resourcemanager-metrics.out is going into current directory instead >>>> of log directory >>>> - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't >>>> even work. Not just deprecated in favor of timelineserver as was advertised. >>>> - Spurious warnings on CLI >>>> 17/11/20 17:07:34 INFO conf.Configuration: >>>> resource-types.xml not found >>>> 17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to >>>> find 'resource-types.xml'. >>>> >>>> Side notes >>>> >>>> - When did we stop putting CHANGES files into the source artifacts? >>>> - Even after "mvn install"ing once, shading is repeated again and >>>> again for every new 'mvn install' even though there are no source changes - >>>> we should see how this can be avoided. >>>> - Compatibility notes >>>> -- NM's env list is curtailed unlike in 2.x (For e.g, >>>> HADOOP_MAPRED_HOME is not automatically inherited. Correct behavior) >>>> -- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar >>>> into hadoop-mapreduce-client-jobclient-3.0.0-tests.jar >>>> >>> >>> Sleep has always been in the jobclient test jar as long as I can >>> remember, so it's not new for 3.0. >>> >>> >>>> >>>> Thanks >>>> +Vinod >>>> >>>> > On Nov 14, 2017, at 1:34 PM, Andrew Wang >>>> wrote: >>>> > >>>> > Hi folks, >>>> > >>>> > Thanks as always to the many, many contributors who helped with this >>>> > release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are >>>> > available here: >>>> > >>>> > http://people.apache.org/~wang/3.0.0-RC0/ >>>> > >>>> > This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific. >>>> > >>>> > 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable >>>> > additions include the merge of YARN resource types, API-based >>>> configuration >>>> > of the CapacityScheduler, and HDFS router-based federation. >>>> > >>>> > I've done my traditional testing with a pseudo cluster and a Pi job. >>>> My +1 >>>> > to start. >>>> > >>>> > Best, >>>> > Andrew >>>> >>>> >>> >> > --001a11411bbeda5947055e77e926--