hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Murthy <...@hortonworks.com>
Subject Re: Hadoop 3.x: what about shipping trunk as a 2.x release in 2015?
Date Mon, 09 Mar 2015 22:54:57 GMT

 Do you have a list of incompatible changes other than the shell-script rewrite? If we do
have others we'd have to fix them anyway for the current plan on hadoop-3.x right? So, I don't
see the difference?


From: Colin P. McCabe <cmccabe@apache.org>
Sent: Monday, March 09, 2015 3:05 PM
To: hdfs-dev@hadoop.apache.org
Cc: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Hadoop 3.x: what about shipping trunk as a 2.x release in 2015?

Java 7 will be end-of-lifed in April 2015.  I think it would be unwise
to plan a new Hadoop release against a version of Java that is almost
obsolete and (soon) no longer receiving security updates.  I think
people will be willing to roll out a new version of Java for Hadoop

Similarly, the whole point of bumping the major version number is the
ability to make incompatible changes.  There are already a bunch of
incompatible changes in the trunk branch.  Are you proposing to revert
those?  Or push them into newly created feature branches?  This
doesn't seem like a good idea to me.

I would be in favor of backporting targetted incompatible changes from
trunk to branch-2.  For example, we could consider pulling in Allen's
shell script rewrite.  But pulling in all of trunk seems like a bad
idea at this point, if we want a 2.x release.


On Mon, Mar 9, 2015 at 2:15 PM, Steve Loughran <stevel@hortonworks.com> wrote:
> If 3.x is going to be Java 8 & not backwards compatible, I don't expect anyone wanting
to use this in production until some time deep into 2016.
> Issue: JDK 8 vs 7
> It will require Hadoop clusters to move up to Java 8. While there's dev pull for this,
there's ops pull against this: people are still in the moving-off Java 6 phase due to that
"it's working, don't update it" philosophy. Java 8 is compelling to us coders, but that doesn't
mean ops want it.
> You can run JDK-8 code in a YARN cluster running on Hadoop 2.7 *today*, the main thing
is setting up JAVA_HOME. That's something we could make easier somehow (maybe some min Java
version field in resource requests that will let apps say java 8, java 9, ...). YARN could
not only set up JVM paths, it could fail-fast if a Java version wasn't available.
> What we can't do in hadoop coretoday  is set javac.version=1.8 & use java 8 code.
Downstream code ca do that (Hive, etc); they just need to accept that they don't get to play
on JDK7 clusters if they embrace l-expressions.
> So...we need to stay on java 7 for some time due to ops pull; downstream apps get to
choose what they want. We can/could enhance YARN to make JVM choice more declarative.
> Issue: Incompatible changes
> Without knowing what is proposed for "an incompatible classpath change", I can't say
whether this is something that could be made optional. If it isn't, then it is a python-3
class option, "rewrite your code" event, which is going to be particularly traumatic to things
like Hive that already do complex CP games. I'm currently against any mandatory change here,
though would love to see an optional one. And if optional, it ceases to become an incompatible
> Issue: Getting trunk out the door
> The main diff from branch-2 and trunk is currently the bash script changes. These don't
break client apps. May or may not break bigtop & other downstream hadoop stacks, but developers
don't need to worry about this:  no recompilation necessary
> Proposed: ship trunk as a 2.x release, compatible with JDK7 & Java code.
> It seems to me that I could go
> git checkout trunk
>         mvn versions:set -DnewVersion=2.8.0-SNAPSHOT
> We'd then have a version of Hadoop-trunk we could ship later this year, compatible at
the JDK and API level with the existing java code & JDK7+ clusters.
> A classpath fix that is optional/compatible can then go out on the 2.x line, saving the
3.x tag for something that really breaks things, forces all downstream apps to set up new
hadoop profiles, have separate modules & generally hate the hadoop dev team
> This lets us tick off the "recent trunk release" and "fixed shell scripts" items, pushing
out those benefits to people sooner rather than later, and puts off the "Hello, we've just
broken your code" event for another 12+ months.
> Comments?
> -Steve

View raw message