From mapreduce-dev-return-18721-apmail-hadoop-mapreduce-dev-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 31 19:33:15 2017 Return-Path: X-Original-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0803B1A467 for ; Thu, 31 Aug 2017 19:33:15 +0000 (UTC) Received: (qmail 62194 invoked by uid 500); 31 Aug 2017 19:33:13 -0000 Delivered-To: apmail-hadoop-mapreduce-dev-archive@hadoop.apache.org Received: (qmail 61958 invoked by uid 500); 31 Aug 2017 19:33:13 -0000 Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list mapreduce-dev@hadoop.apache.org Received: (qmail 61916 invoked by uid 99); 31 Aug 2017 19:33:13 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Aug 2017 19:33:13 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 998A6C3544; Thu, 31 Aug 2017 19:33:12 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.401 X-Spam-Level: X-Spam-Status: No, score=-0.401 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.8, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 5NhvKCq0jmEK; Thu, 31 Aug 2017 19:33:09 +0000 (UTC) Received: from mail-vk0-f47.google.com (mail-vk0-f47.google.com [209.85.213.47]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id E8B325F21F; Thu, 31 Aug 2017 19:33:08 +0000 (UTC) Received: by mail-vk0-f47.google.com with SMTP id x85so1627334vkx.5; Thu, 31 Aug 2017 12:33:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=+6CboW0sZt0ZfeCw06CboVhDgFTXAdfbDIxCx0+mIWE=; b=G6dqHCLPjkbPPwFaDKmdXKt/bJ+GTbbmu4D0XNSFrGD13pJWaLnTdkbWRHiQs8g4xn Df2xTjyNZTHUmH+iNPKzTLYdPCoIHshptBGP4VZgZEVx0jsxqDFHEw95lgcBLKTKI3hz FdzijRXjSIWmZHUGTk3XEirSqvk3ufGp5Yycp4eSghFJ7r6z4eDUPElknFoL+SZFhkRF gljM1WXX2NBRLODt1xvtvRqda1DfgqdHOd69faRCYDxrimxC5SKw3Q1G5FuKz9BobrhT ME+2fnrI8VxA210XL+Gp9fMTgv+9bmK9ky2w7ie5plQGiU9yRe37fGVgFisl/dNR7GoH WljQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=+6CboW0sZt0ZfeCw06CboVhDgFTXAdfbDIxCx0+mIWE=; b=B8+06LdTOkWPQCOTXk12jt4mLO2Z5elx3SpM85uFOKYRSjCIRywGA+VyTnE8xiQwoc /N5F2ohMaZhBNTh4Mab+IPCb9HrOfKrUBzi1EJiVKp0+d6H/k67DmaFUdBrd9ziaTIpL MP/gaDLy+HJMim/Y6mfpO5xYpQAErRvZyCj9ieby5r5YANuwQEGfgFWlVBZOOZQnxioS UFivgSttn9OnMd5p/OjO3kQpfOAc51/vgzdASxBCg/LP8DdW9bKgaO+AobTIwvWdgI7f x2vdUVSf+1ar+k/AKhunHZSonfuduurTBF6LW+8ZiMc8UftJnjDg/4rYoIDUuS1aBuK0 S9fg== X-Gm-Message-State: AHYfb5jZFE0ELnvmQCTukQ/zIf4pRtRGGQ303x/VQwC2K2X7yTl3zrYb oQMY2x8IdWqRI2hZ/rSlfQe/jrRSAQ== X-Google-Smtp-Source: ADKCNb61tuA7C6qPND3v/S62NGjXUP5iS+yRmnIWAlLP/L4OeAXV3Qqhy5F+VTFEC+sirP2Q6w9b6ImsEU9QnHxpWBM= X-Received: by 10.31.195.195 with SMTP id t186mr4057897vkf.162.1504207981658; Thu, 31 Aug 2017 12:33:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.103.135.197 with HTTP; Thu, 31 Aug 2017 12:33:01 -0700 (PDT) In-Reply-To: References: From: Subramaniam V K Date: Thu, 31 Aug 2017 12:33:01 -0700 Message-ID: Subject: Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk To: Ravi Prakash Cc: varunsaxena@apache.org, Vrushali C , Sangjin Lee , Steve Loughran , "common-dev@hadoop.apache.org" , "yarn-dev@hadoop.apache.org" , "mapreduce-dev@hadoop.apache.org" , "hdfs-dev@hadoop.apache.org" Content-Type: multipart/alternative; boundary="001a114e6bc86a4739055811b6c8" --001a114e6bc86a4739055811b6c8 Content-Type: text/plain; charset="UTF-8" Good to see this merged. I have initiated a separate thread with a smaller set of stakeholders to discuss inclusion in 2.9. We'll report back to the 2.9 release thread as soon as we reach consensus. On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash wrote: > +1 to maintaining history. > > On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org < > varun.saxena.apache@gmail.com> wrote: > > > Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk. > > Maintaining history I believe can be useful as it can make reverts > > easier if at all required. > > And can be an easy reference point to look at who had contributed what > > without having to go back to the branch. > > > > Regards, > > Varun Saxena. > > > > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C > > wrote: > > > > > Thanks Sangjin for the link to the previous discussions on this! I > think > > > that helps answer Steve's questions. > > > > > > As decided on that thread [1], YARN-5355 as a feature branch was merged > > to > > > trunk via "git merge --no-ff" . > > > > > > Although trunk already had TSv2 code (alpha1) prior to this merge, we > > > chose to develop on a feature branch YARN-5355 so that we could control > > > when changes went into trunk and didn't inadvertently disrupt trunk. > > > > > > Is the latest merge causing any conflicts or issues for s3guard, Steve? > > > > > > thanks > > > Vrushali > > > [1] https://lists.apache.org/thread.html/ > 43cd65c6b6c3c0e8ac2b3c76afd9ef > > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org > %3E > > > > > > > > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee wrote: > > > > > >> I recall this discussion about a couple of years ago: > > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac > > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon- > > >> dev.hadoop.apache.org%3E > > >> > > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran < > stevel@hortonworks.com > > > > > >> wrote: > > >> > > >>> I'd have assumed it would have gone in as one single patch, rather > than > > >>> a full history. I don't see why the trunk needs all the evolutionary > > >>> history of a build. > > >>> > > >>> What should our policy/process be here? > > >>> > > >>> I do currently plan to merge the s3guard in as one single squashed > > >>> patch; just getting HADOOP-14809 sorted first. > > >>> > > >>> > > >>> > On 30 Aug 2017, at 07:09, Vrushali C > > wrote: > > >>> > > > >>> > I'm adding my +1 (binding) to conclude the vote. > > >>> > > > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get > on > > >>> with > > >>> > the merge to trunk shortly. Thanks everyone! > > >>> > > > >>> > Regards > > >>> > Vrushali > > >>> > > > >>> > > > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org < > > >>> > varun.saxena.apache@gmail.com> wrote: > > >>> > > > >>> >> +1 (binding). > > >>> >> > > >>> >> Kudos to all the team members for their great work! > > >>> >> > > >>> >> Being part of the ATSv2 team, I have been involved with either > > >>> development > > >>> >> or review of most of the JIRAs'. > > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified > that > > >>> there > > >>> >> is no impact when ATSv2 is turned off. > > >>> >> > > >>> >> Regards, > > >>> >> Varun Saxena. > > >>> >> > > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan < > > >>> >> vrushalic2016@gmail.com> wrote: > > >>> >> > > >>> >>> Hi folks, > > >>> >>> > > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to > > merge > > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The > > >>> vote > > >>> >>> will > > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT. > > >>> >>> > > >>> >>> We have previously completed one merge onto trunk [3] and > Timeline > > >>> Service > > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1. > > >>> >>> > > >>> >>> Since then, we have been working on extending the capabilities of > > >>> Timeline > > >>> >>> Service v2 in a feature branch [2] for a while, and we are > > reasonably > > >>> >>> confident that the state of the feature meets the criteria to be > > >>> merged > > >>> >>> onto trunk and we'd love folks to get their hands on it in a test > > >>> capacity > > >>> >>> and provide valuable feedback so that we can make it > > >>> production-ready. > > >>> >>> > > >>> >>> In a nutshell, Timeline Service v.2 delivers significant > > scalability > > >>> and > > >>> >>> usability improvements based on a new architecture. What we would > > >>> like to > > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature > has a > > >>> >>> complete end-to-end read/write flow with security and read level > > >>> >>> authorization via whitelists. You should be able to start setting > > it > > >>> up > > >>> >>> and > > >>> >>> testing it. > > >>> >>> > > >>> >>> At a high level, the following are the key features that have > been > > >>> >>> implemented since alpha1: > > >>> >>> - Security via Kerberos Authentication and delegation tokens > > >>> >>> - Read side simple authorization via whitelist > > >>> >>> - Client configurable entity sort ordering > > >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching > > >>> metrics by > > >>> >>> timerange, pagination, sub-app entities > > >>> >>> - Support for storing sub-application entities (entities that > exist > > >>> >>> outside > > >>> >>> the scope of an application) > > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable table > > >>> >>> prefixes, > > >>> >>> configurable hbase cluster > > >>> >>> - Flow level aggregations done as dynamic (table level) > > coprocessors > > >>> >>> - Uses latest stable HBase release 1.2.6 > > >>> >>> > > >>> >>> There are a total of 82 subtasks that were completed as part of > > this > > >>> >>> effort. > > >>> >>> > > >>> >>> We paid close attention to ensure that once disabled Timeline > > >>> Service v.2 > > >>> >>> does not impact existing functionality when disabled (by > default). > > >>> >>> > > >>> >>> Special thanks to a team of folks who worked hard and contributed > > >>> towards > > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma K > S, > > >>> Varun > > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, > > Joep > > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack. > > >>> >>> > > >>> >>> Regards, > > >>> >>> Vrushali > > >>> >>> > > >>> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27 > > >>> 383.html > > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355 > > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928 > > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355 > > >>> >>> > > >>> >> > > >>> >> > > >>> > > >>> > > >>> ------------------------------------------------------------ > --------- > > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org > > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org > > >>> > > >>> > > >> > > > > > > --001a114e6bc86a4739055811b6c8--