spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Luciano Resende <luckbr1...@gmail.com>
Subject Re: [discuss] ending support for Java 7 in Spark 2.0
Date Mon, 04 Apr 2016 15:48:38 GMT
Reynold,

Considering the performance improvements you mentioned in your original
e-mail and also considering that few other big data projects have already
or are in progress of abandoning JDK 7, I think it would benefit Spark if
we go with JDK 8.0 only.

Are there users that will be less aggressive ? Yes, but those would most
likely be in more stable releases like 1.6.x.

On Sun, Apr 3, 2016 at 10:28 PM, Reynold Xin <rxin@databricks.com> wrote:

> Since my original email, I've talked to a lot more users and looked at
> what various environments support. It is true that a lot of enterprises,
> and even some technology companies, are still using Java 7. One thing is
> that up until this date, users still can't install openjdk 8 on Ubuntu by
> default. I see that as an indication that it is too early to drop Java 7.
>
> Looking at the timeline, JDK release a major new version roughly every 3
> years. We dropped Java 6 support one year ago, so from a timeline point of
> view we would be very aggressive here if we were to drop Java 7 support in
> Spark 2.0.
>
> Note that not dropping Java 7 support now doesn't mean we have to support
> Java 7 throughout Spark 2.x. We dropped Java 6 support in Spark 1.5, even
> though Spark 1.0 started with Java 6.
>
> In terms of testing, Josh has actually improved our test infra so now we
> would run the Java 8 tests: https://github.com/apache/spark/pull/12073
>
>
>
>
> On Thu, Mar 24, 2016 at 8:51 PM, Liwei Lin <lwlin7@gmail.com> wrote:
>
>> Arguments are really convincing; new Dataset API as well as performance
>>
>> improvements is exiting, so I'm personally +1 on moving onto Java8.
>>
>>
>>
>> However, I'm afraid Tencent is one of "the organizations stuck with
>> Java7"
>>
>> -- our IT Infra division wouldn't upgrade to Java7 until Java8 is out, and
>>
>> wouldn't upgrade to Java8 until Java9 is out.
>>
>>
>> So:
>>
>> (non-binding) +1 on dropping scala 2.10 support
>>
>> (non-binding)  -1 on dropping Java 7 support
>>
>>                       * as long as we figure out a practical way to run
>> Spark with
>>
>>                         JDK8 on JDK7 clusters, this -1 would then
>> definitely be +1
>>
>>
>> Thanks !
>>
>> On Fri, Mar 25, 2016 at 10:28 AM, Koert Kuipers <koert@tresata.com>
>> wrote:
>>
>>> i think that logic is reasonable, but then the same should also apply to
>>> scala 2.10, which is also unmaintained/unsupported at this point (basically
>>> has been since march 2015 except for one hotfix due to a license
>>> incompatibility)
>>>
>>> who wants to support scala 2.10 three years after they did the last
>>> maintenance release?
>>>
>>>
>>> On Thu, Mar 24, 2016 at 9:59 PM, Mridul Muralidharan <mridul@gmail.com>
>>> wrote:
>>>
>>>> Removing compatibility (with jdk, etc) can be done with a major
>>>> release- given that 7 has been EOLed a while back and is now unsupported,
>>>> we have to decide if we drop support for it in 2.0 or 3.0 (2+ years from
>>>> now).
>>>>
>>>> Given the functionality & performance benefits of going to jdk8, future
>>>> enhancements relevant in 2.x timeframe ( scala, dependencies) which
>>>> requires it, and simplicity wrt code, test & support it looks like a
good
>>>> checkpoint to drop jdk7 support.
>>>>
>>>> As already mentioned in the thread, existing yarn clusters are
>>>> unaffected if they want to continue running jdk7 and yet use
>>>> spark2 (install jdk8 on all nodes and use it via JAVA_HOME, or worst case
>>>> distribute jdk8 as archive - suboptimal).
>>>> I am unsure about mesos (standalone might be easier upgrade I guess ?).
>>>>
>>>>
>>>> Proposal is for 1.6x line to continue to be supported with critical
>>>> fixes; newer features will require 2.x and so jdk8
>>>>
>>>> Regards
>>>> Mridul
>>>>
>>>>
>>>> On Thursday, March 24, 2016, Marcelo Vanzin <vanzin@cloudera.com>
>>>> wrote:
>>>>
>>>>> On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin <rxin@databricks.com>
>>>>> wrote:
>>>>> > If you want to go down that route, you should also ask somebody
who
>>>>> has had
>>>>> > experience managing a large organization's applications and try
to
>>>>> update
>>>>> > Scala version.
>>>>>
>>>>> I understand both sides. But if you look at what I've been asking
>>>>> since the beginning, it's all about the cost and benefits of dropping
>>>>> support for java 1.7.
>>>>>
>>>>> The biggest argument in your original e-mail is about testing. And the
>>>>> testing cost is much bigger for supporting scala 2.10 than it is for
>>>>> supporting java 1.7. If you read one of my earlier replies, it should
>>>>> be even possible to just do everything in a single job - compile for
>>>>> java 7 and still be able to test things in 1.8, including lambdas,
>>>>> which seems to be the main thing you were worried about.
>>>>>
>>>>>
>>>>> > On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin <vanzin@cloudera.com>
>>>>> wrote:
>>>>> >>
>>>>> >> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin <rxin@databricks.com>
>>>>> wrote:
>>>>> >> > Actually it's *way* harder to upgrade Scala from 2.10 to
2.11,
>>>>> than
>>>>> >> > upgrading the JVM runtime from 7 to 8, because Scala 2.10
and
>>>>> 2.11 are
>>>>> >> > not
>>>>> >> > binary compatible, whereas JVM 7 and 8 are binary compatible
>>>>> except
>>>>> >> > certain
>>>>> >> > esoteric cases.
>>>>> >>
>>>>> >> True, but ask anyone who manages a large cluster how long it
would
>>>>> >> take them to upgrade the jdk across their cluster and validate
all
>>>>> >> their applications and everything... binary compatibility is
a tiny
>>>>> >> drop in that bucket.
>>>>> >>
>>>>> >> --
>>>>> >> Marcelo
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Marcelo
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>
>>>>>
>>>
>>
>


-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Mime
View raw message