hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Travis Thompson <travis.r.thomp...@gmail.com>
Subject Re: In hindsight... Re: Thinking ahead to hadoop-2.6
Date Wed, 17 Sep 2014 22:33:37 GMT
There's actually an umbrella JIRA to track issues with JDK8
(HADOOP-11090), in case anyone missed it.

At LinkedIn we've been running our Hadoop 2.3 deployment on JDK8 for
about a month now with some mixed results.  It definitely works but
there are issues, mostly around virtual memory exploding.  The reason
we took the jump early is there is a company wide push to move to JDK8
ASAP, I suspect this isn't something unique to LinkedIn.   To get this
to work with security enabled, we've had to apply patches not even in
trunk yet because they break JDK6 compatibility.

>From my perspective, based on what I've seen and people I've talked
to, there is a huge push to move to JDK8 ASAP so it's becoming
increasingly urgent to at least get support to run on JDK8.

On Wed, Sep 17, 2014 at 9:55 AM, Allen Wittenauer <aw@altiscale.com> wrote:
> On Sep 17, 2014, at 2:47 AM, Steve Loughran <stevel@hortonworks.com> wrote:
>> I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down the
>> filesystem binding with more tests than ever before.
>         FWIW, based upon my survey of JIRA, there are a lot of unit test fixes that are
only in trunk.
>> But I am also aware of large organisations that are still on Java 6.
>> Giving a clear roadmap "move to Java 7 now, java 8 in "XX months" can help
>> them plan.
>         Planning is a big thing.  That’s one of the reasons why it’d be prudent to
start doing 3.0+JDK8 now as well.  Even if April slips, other projects and orgs are already
moving to 8.  These people wonder what our plans are so that they can run one JVM.  Right
now our answer is ¯\_(ツ)_/¯ .
>         I’m sure I can dig up a user running Hadoop 0.13 because it ran on JDK5.  That
doesn’t mean the open source project should stall because certain orgs don’t/can't upgrade.
>>>        Drop the 2.6.0 release, branch trunk, and start rolling a
>>> 3.0.0-alpha with JDK8 as the minimum.  2.5.1 becomes the base for all
>>> sustaining work.  This gives the rest of the community time to move to JDK8
>>> if they haven’t already.  For downstream vendors, it gives a roadmap for
>>> their customers who will be asking about JDK8 sooner rather than later.  By
>>> the time 3.0 stabilizes, we’re probably looking at April, which is perfect
>>> timing.
>> That delays getting stuff out too much; if april slips it becomes a long
>> time since an ASF release came out.
>         I’m assuming you specifically mean a ‘stable’ release.  If, as everyone
seems to be saying, that 3.x doesn’t have that much different than 2.x, doesn’t this mean
that 3.x should be stable much quicker than 2.x took?  In other words, if 2.5 is stable and
the biggest differences between it and trunk is the majority of code (450+ JIRAs as of yesterday
afternoon) that just also happens to be in 2.6, doesn’t it mean 2.6 is also extremely unstable?
 (Thus supporting my conjecture that 2.6 is going to be a problematic release?)
>> Saying "you must run on Java 8 for
>> this" will only scare people off and hold back adoption of 3.x, leaving 2.5
>> as the last release that ends up being used for a while; the new 1.0.4
>         From the outside, trunk looks a lot of 0.21 already.  From what I can tell, there
is zero motivation to get it out the door and on a roadmap. Primarily because there is little
different between trunk and branch-2.  This is a very dangerous place to be as those few differences,
some measured in years old, rot and wither. :(
>> Here's an alternative
>> -2.6 on java 6, announce EOL for Java 6 support
>> -2.7 on Java 7, state that the lifespan of j7 support will be some bounded
>> time period (12-18 mo)
>> -trunk to build and test on Java 8 in jenkins alongside java 7. For that to
>> be useful someone needs to volunteer to care about build failures. are you
>> volunteering, Allen?
>         This seems reasonable, except what release should folks who *require* java 8
use? Nightly trunk+patches builds? How do downstream projects test?  Should JDK8 fixes be
going into a branch?  (I’m making the assumption that fixes for JDK8 are not backward compatible
with JDK7.  Hopefully they are, but given our usage of private APIs…)
>         I’ve been approached by a few people over the past month+ if I’d be interested
in or will be RM’ing 3.0.  I’m seriously considering it esp given a) it’d be a nice
learning experience for me  b) my “day job” makes it practical time-wise c) I seem to
be the only one concerned enough about quite a bit of stale code  to get it out the door.
>         FWIW, I’m in the process of moving my test vm to JDK8 to see how bad the damage
truly is right now. Based on others, it seems security doesn’t work, which is a pretty big
deal.  I can certainly start posting trunk builds on people.apache.org if folks are interested.
>> -we switch trunk to Java 7 NOW. That doesn't mean a rewrite fest going
>> through all catch() statements making them multicatch, and the same for
>> string case.
>         Yup.  There’s little reason *not* to switch trunk to JDK7 now.

View raw message