From mapreduce-dev-return-19096-apmail-hadoop-mapreduce-dev-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 7 19:09:10 2017 Return-Path: X-Original-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A38C7173EE for ; Tue, 7 Nov 2017 19:09:10 +0000 (UTC) Received: (qmail 76340 invoked by uid 500); 7 Nov 2017 19:09:06 -0000 Delivered-To: apmail-hadoop-mapreduce-dev-archive@hadoop.apache.org Received: (qmail 76139 invoked by uid 500); 7 Nov 2017 19:09:06 -0000 Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list mapreduce-dev@hadoop.apache.org Received: (qmail 76088 invoked by uid 99); 7 Nov 2017 19:09:06 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Nov 2017 19:09:06 +0000 Received: from [10.22.9.120] (unknown [192.175.27.10]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 99EA01A00A5; Tue, 7 Nov 2017 19:09:04 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\)) Subject: Re: [DISCUSS] A final minor release off branch-2? From: Vinod Kumar Vavilapalli In-Reply-To: <8D5F7E3237B3ED47B84CF187BB17B66662BB22A3@SHSMSX103.ccr.corp.intel.com> Date: Tue, 7 Nov 2017 11:08:49 -0800 Cc: Andrew Wang , Arun Suresh , "common-dev@hadoop.apache.org" , "yarn-dev@hadoop.apache.org" , Hdfs-dev , "mapreduce-dev@hadoop.apache.org" Content-Transfer-Encoding: quoted-printable Message-Id: <582E3AE2-E314-4378-8F26-564A49F8E405@apache.org> References: <8D5F7E3237B3ED47B84CF187BB17B66662BB22A3@SHSMSX103.ccr.corp.intel.com> To: "Zheng, Kai" X-Mailer: Apple Mail (2.3445.4.7) Thanks for your comments, Zheng. Replies inline. > On the other hand, I've discussed with quite a few 3.0 potential = users, it looks like most of them are interested in the erasure coding = feature and a major scenario for that is to back up their large volume = of data to save storage cost. They might run analytics workload using = Hive, Spark, Impala and Kylin on the new cluster based on the version, = but it's not a must at the first time. They understand there might be = some gaps so they'd migrate their workloads incrementally. For the major = analytics workload, we've performed lots of benchmark and integration = tests as well as other sides I believe, we did find some issues but they = should be fixed in downstream projects. I thought the release of GA will = accelerate the progress and expose the issues if any. We couldn't wait = for it being matured. There isn't perfectness. 3.0 is a GA release from the Apache Hadoop community. So, we cannot = assume that all usages in the short term are *only* going to be for = storage optimization features and only on dedicated clusters. We have to = make sure that the workloads can be migrated right now and/or that = existing clusters can be upgraded in-place. If not, we shouldn't be = calling it GA. > This sounds a good consideration. I'm thinking if I'm a Hadoop user, = for example, I'm using 2.7.4 or 2.8.2 or whatever 2.x version, would I = first upgrade to this bridging release then use the bridge support to = upgrade to 3.x version? I'm not sure. On the other hand, I might tend to = look for some guides or supports in 3.x docs about how to upgrade from = 2.7 to 3.x.=20 Arun Suresh also asked this same question earlier. I think this will = really depend on what we discover as part of the migration and = user-acceptance testing. If we don't find major issues, you are right, = folks can jump directly from one of 2.7, 2.8 or 2.9 to 3.0. > Frankly speaking, working on some bridging release not targeting any = feature isn't so attractive to me as a contributor. Overall, the final = minor release off branch-2 is good, we should also give 3.x more time to = evolve and mature, therefore it looks to me we would have to work on two = release lines meanwhile for some time. I'd like option C), and suggest = we focus on the recent releases. Answering this question is also one of the goals of my starting this = thread. Collectively we need to conclude if we are okay or not okay with = no longer putting any new feature work in general on the 2.x line after = 2.9.0 release and move over our focus into 3.0. Thanks +Vinod > -----Original Message----- > From: Vinod Kumar Vavilapalli [mailto:vinodkv@apache.org]=20 > Sent: Tuesday, November 07, 2017 9:43 AM > To: Andrew Wang > Cc: Arun Suresh ; common-dev@hadoop.apache.org; = yarn-dev@hadoop.apache.org; Hdfs-dev ; = mapreduce-dev@hadoop.apache.org > Subject: Re: [DISCUSS] A final minor release off branch-2? >=20 > The main goal of the bridging release is to ease transition on stuff = that is guaranteed to be broken. >=20 > Of the top of my head, one of the biggest areas is application = compatibility. When folks move from 2.x to 3.x, are their apps binary = compatible? Source compatible? Or need changes? >=20 > In 1.x -> 2.x upgrade, we did a bunch of work to atleast make old apps = be source compatible. This means relooking at the API compatibility in = 3.x and their impact of migrating applications. We will have to revist = and un-deprecate old APIs, un-delete old APIs and write documentation on = how apps can be migrated. >=20 > Most of this work will be in 3.x line. The bridging release on the = other hand will have deprecation for APIs that cannot be undeleted. This = may be already have been done in many places. But we need to make sure = and fill gaps if any. >=20 > Other areas that I can recall from the old days > - Config migration: Many configs are deprecated or deleted. We need = documentation to help folks to move. We also need deprecations in the = bridging release for configs that cannot be undeleted. > - You mentioned rolling-upgrades: It will be good to exactly outline = the type of testing. For e.g., the rolling-upgrades orchestration order = has direct implication on the testing done. > - Story for downgrades? > - Copying data between 2.x clusters and 3.x clusters: Does this work = already? Is it broken anywhere that we cannot fix? Do we need bridging = features for this work? >=20 > +Vinod >=20 >> On Nov 6, 2017, at 12:49 PM, Andrew Wang = wrote: >>=20 >> What are the known gaps that need bridging between 2.x and 3.x? >>=20 >> =46rom an HDFS perspective, we've tested wire compat, rolling = upgrade,=20 >> and rollback. >>=20 >> =46rom a YARN perspective, we've tested wire compat and rolling = upgrade.=20 >> Arun just mentioned an NM rollback issue that I'm not familiar with. >>=20 >> Anything else? External to this discussion, these should be = documented=20 >> as known issues for 3.0. >>=20 >> Best. >> Andrew >>=20 >> On Sun, Nov 5, 2017 at 1:46 PM, Arun Suresh = wrote: >>=20 >>> Thanks for starting this discussion VInod. >>>=20 >>> I agree (C) is a bad idea. >>> I would prefer (A) given that ATM, branch-2 is still very close to >>> branch-2.9 - and it is a good time to make a collective decision to=20= >>> lock down commits to branch-2. >>>=20 >>> I think we should also clearly define what the 'bridging' release=20 >>> should be. >>> I assume it means the following: >>> * Any 2.x user wanting to move to 3.x must first upgrade to the=20 >>> bridging release first and then upgrade to the 3.x release. >>> * With regard to state store upgrades (at least NM state stores) the=20= >>> bridging state stores should be aware of all new 3.x keys so the=20 >>> implicit assumption would be that a user can only rollback from the=20= >>> 3.x release to the bridging release and not to the old 2.x release. >>> * Use the opportunity to clean up deprecated API ? >>> * Do we even want to consider a separate bridging release for 2.7,=20= >>> 2.8 an >>> 2.9 lines ? >>>=20 >>> Cheers >>> -Arun >>>=20 >>> On Fri, Nov 3, 2017 at 5:07 PM, Vinod Kumar Vavilapalli <=20 >>> vinodkv@apache.org> >>> wrote: >>>=20 >>>> Hi all, >>>>=20 >>>> With 3.0.0 GA around the corner (tx for the push, Andrew!), 2.9.0 = RC=20 >>>> out (tx Arun / Subru!) and 2.8.2 (tx Junping!), I think it's high=20= >>>> time we >>> have >>>> a discussion on how we manage our developmental bandwidth between=20= >>>> 2.x >>> line >>>> and 3.x lines. >>>>=20 >>>> Once 3.0 GA goes out, we will have two parallel and major release = lines. >>>> The last time we were in this situation was back when we did 1.x ->=20= >>>> 2.x jump. >>>>=20 >>>> The parallel releases implies overhead of decisions, branch-merges=20= >>>> and back-ports. Right now we already do backports for 2.7.5, 2.8.2,=20= >>>> 2.9.1, >>>> 3.0.1 and potentially a 3.1.0 in a few months after 3.0.0 GA. And=20= >>>> many of these lines - for e.g 2.8, 2.9 - are going to be used for a=20= >>>> while at a bunch of large sites! At the same time, our users won't=20= >>>> migrate to 3.0 GA overnight - so we do have to support two parallel = lines. >>>>=20 >>>> I propose we start thinking of the fate of branch-2. The idea is to=20= >>>> have one final release that helps our users migrate from 2.x to = 3.x.=20 >>>> This includes any changes on the older line to bridge compatibility=20= >>>> issues, upgrade issues, layout changes, tooling etc. >>>>=20 >>>> We have a few options I think >>>> (A) >>>> -- Make 2.9.x the last minor release off branch-2 >>>> -- Have a maintenance release that bridges 2.9 to 3.x >>>> -- Continue to make more maintenance releases on 2.8 and 2.9 as=20= >>>> necessary >>>> -- All new features obviously only go into the 3.x line as no >>> features >>>> can go into the maint line. >>>>=20 >>>> (B) >>>> -- Create a new 2.10 release which doesn't have any new features,=20= >>>> but as a bridging release >>>> -- Continue to make more maintenance releases on 2.8, 2.9 and=20 >>>> 2.10 as necessary >>>> -- All new features, other than the bridging changes, go into the=20= >>>> 3.x line >>>>=20 >>>> (C) >>>> -- Continue making branch-2 releases and postpone this discussion=20= >>>> for later >>>>=20 >>>> I'm leaning towards (A) or to a lesser extent (B). Willing to hear=20= >>>> otherwise. >>>>=20 >>>> Now, this obviously doesn't mean blocking of any more minor = releases=20 >>>> on branch-2. Obviously, any interested committer / PMC can roll up=20= >>>> his/her sleeves, create a release plan and release, but we all need=20= >>>> to >>> acknowledge >>>> that versions are not cheap and figure out how the community=20 >>>> bandwidth is split overall. >>>>=20 >>>> Thanks >>>> +Vinod >>>> PS: The proposal is obviously not to force everyone to go in one >>> direction >>>> but more of a nudging the community to figure out if we can focus a=20= >>>> major part of of our bandwidth on one line. I had a similar concern=20= >>>> when we >>> were >>>> doing 2.8 and 3.0 in parallel, but the impending possibility of=20 >>>> spreading too thin is much worse IMO. >>>> PPS: (C) is a bad choice. With 2.8 and 2.9 we are already seeing=20 >>>> user adoption splintering between two lines. With 2.10, 2.11 etc=20 >>>> coexisting >>> with >>>> 3.0, 3.1 etc, we will revisit the mad phase years ago when we had=20= >>>> 0.20.x, 0.20-security coexisting with 0.21, 0.22 etc. >>>=20 >=20 >=20 > --------------------------------------------------------------------- > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org > For additional commands, e-mail: common-dev-help@hadoop.apache.org >=20 --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org