spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karlis Zigurs <homolu...@gmail.com>
Subject Re: [discuss] ending support for Java 7 in Spark 2.0
Date Mon, 04 Apr 2016 22:01:20 GMT
Curveball: Is there a need to use lambdas quite yet?

On Mon, Apr 4, 2016 at 10:58 PM, Ofir Manor <ofir.manor@equalum.io> wrote:
> I think that a backup plan could be to announce that JDK7 is deprecated in
> Spark 2.0 and support for it will be fully removed in Spark 2.1. This gives
> admins enough warning to install JDK8 along side their "main" JDK (or fully
> migrate to it), while allowing the project to merge JDK8-specific changes to
> trunk right after the 2.0 release.
>
> However, I personally think it is better to drop JDK7 now. I'm sure that
> both the community and the distributors (Databricks, Cloudera, Hortonworks,
> MapR, IBM etc) will all rush to help their customers migrate their
> environment to support Spark 2.0, so I think any backlash won't be dramatic
> or lasting.
>
> Just my two cents,
>
> Ofir Manor
>
> Co-Founder & CTO | Equalum
>
> Mobile: +972-54-7801286 | Email: ofir.manor@equalum.io
>
>
> On Mon, Apr 4, 2016 at 6:48 PM, Luciano Resende <luckbr1975@gmail.com>
> wrote:
>>
>> Reynold,
>>
>> Considering the performance improvements you mentioned in your original
>> e-mail and also considering that few other big data projects have already or
>> are in progress of abandoning JDK 7, I think it would benefit Spark if we go
>> with JDK 8.0 only.
>>
>> Are there users that will be less aggressive ? Yes, but those would most
>> likely be in more stable releases like 1.6.x.
>>
>> On Sun, Apr 3, 2016 at 10:28 PM, Reynold Xin <rxin@databricks.com> wrote:
>>>
>>> Since my original email, I've talked to a lot more users and looked at
>>> what various environments support. It is true that a lot of enterprises, and
>>> even some technology companies, are still using Java 7. One thing is that up
>>> until this date, users still can't install openjdk 8 on Ubuntu by default. I
>>> see that as an indication that it is too early to drop Java 7.
>>>
>>> Looking at the timeline, JDK release a major new version roughly every 3
>>> years. We dropped Java 6 support one year ago, so from a timeline point of
>>> view we would be very aggressive here if we were to drop Java 7 support in
>>> Spark 2.0.
>>>
>>> Note that not dropping Java 7 support now doesn't mean we have to support
>>> Java 7 throughout Spark 2.x. We dropped Java 6 support in Spark 1.5, even
>>> though Spark 1.0 started with Java 6.
>>>
>>> In terms of testing, Josh has actually improved our test infra so now we
>>> would run the Java 8 tests: https://github.com/apache/spark/pull/12073
>>>
>>>
>>>
>>>
>>> On Thu, Mar 24, 2016 at 8:51 PM, Liwei Lin <lwlin7@gmail.com> wrote:
>>>>
>>>> Arguments are really convincing; new Dataset API as well as performance
>>>>
>>>> improvements is exiting, so I'm personally +1 on moving onto Java8.
>>>>
>>>>
>>>>
>>>> However, I'm afraid Tencent is one of "the organizations stuck with
>>>> Java7"
>>>>
>>>> -- our IT Infra division wouldn't upgrade to Java7 until Java8 is out,
>>>> and
>>>>
>>>> wouldn't upgrade to Java8 until Java9 is out.
>>>>
>>>>
>>>> So:
>>>>
>>>> (non-binding) +1 on dropping scala 2.10 support
>>>>
>>>> (non-binding)  -1 on dropping Java 7 support
>>>>
>>>>                       * as long as we figure out a practical way to run
>>>> Spark with
>>>>
>>>>                         JDK8 on JDK7 clusters, this -1 would then
>>>> definitely be +1
>>>>
>>>>
>>>> Thanks !
>>>>
>>>>
>>>> On Fri, Mar 25, 2016 at 10:28 AM, Koert Kuipers <koert@tresata.com>
>>>> wrote:
>>>>>
>>>>> i think that logic is reasonable, but then the same should also apply
>>>>> to scala 2.10, which is also unmaintained/unsupported at this point
>>>>> (basically has been since march 2015 except for one hotfix due to a license
>>>>> incompatibility)
>>>>>
>>>>> who wants to support scala 2.10 three years after they did the last
>>>>> maintenance release?
>>>>>
>>>>>
>>>>> On Thu, Mar 24, 2016 at 9:59 PM, Mridul Muralidharan <mridul@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Removing compatibility (with jdk, etc) can be done with a major
>>>>>> release- given that 7 has been EOLed a while back and is now unsupported,
we
>>>>>> have to decide if we drop support for it in 2.0 or 3.0 (2+ years
from now).
>>>>>>
>>>>>> Given the functionality & performance benefits of going to jdk8,
>>>>>> future enhancements relevant in 2.x timeframe ( scala, dependencies)
which
>>>>>> requires it, and simplicity wrt code, test & support it looks
like a good
>>>>>> checkpoint to drop jdk7 support.
>>>>>>
>>>>>> As already mentioned in the thread, existing yarn clusters are
>>>>>> unaffected if they want to continue running jdk7 and yet use spark2
(install
>>>>>> jdk8 on all nodes and use it via JAVA_HOME, or worst case distribute
jdk8 as
>>>>>> archive - suboptimal).
>>>>>> I am unsure about mesos (standalone might be easier upgrade I guess
>>>>>> ?).
>>>>>>
>>>>>>
>>>>>> Proposal is for 1.6x line to continue to be supported with critical
>>>>>> fixes; newer features will require 2.x and so jdk8
>>>>>>
>>>>>> Regards
>>>>>> Mridul
>>>>>>
>>>>>>
>>>>>> On Thursday, March 24, 2016, Marcelo Vanzin <vanzin@cloudera.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin <rxin@databricks.com>
>>>>>>> wrote:
>>>>>>> > If you want to go down that route, you should also ask somebody
who
>>>>>>> > has had
>>>>>>> > experience managing a large organization's applications
and try to
>>>>>>> > update
>>>>>>> > Scala version.
>>>>>>>
>>>>>>> I understand both sides. But if you look at what I've been asking
>>>>>>> since the beginning, it's all about the cost and benefits of
dropping
>>>>>>> support for java 1.7.
>>>>>>>
>>>>>>> The biggest argument in your original e-mail is about testing.
And
>>>>>>> the
>>>>>>> testing cost is much bigger for supporting scala 2.10 than it
is for
>>>>>>> supporting java 1.7. If you read one of my earlier replies, it
should
>>>>>>> be even possible to just do everything in a single job - compile
for
>>>>>>> java 7 and still be able to test things in 1.8, including lambdas,
>>>>>>> which seems to be the main thing you were worried about.
>>>>>>>
>>>>>>>
>>>>>>> > On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin
>>>>>>> > <vanzin@cloudera.com> wrote:
>>>>>>> >>
>>>>>>> >> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin <rxin@databricks.com>
>>>>>>> >> wrote:
>>>>>>> >> > Actually it's *way* harder to upgrade Scala from
2.10 to 2.11,
>>>>>>> >> > than
>>>>>>> >> > upgrading the JVM runtime from 7 to 8, because
Scala 2.10 and
>>>>>>> >> > 2.11 are
>>>>>>> >> > not
>>>>>>> >> > binary compatible, whereas JVM 7 and 8 are binary
compatible
>>>>>>> >> > except
>>>>>>> >> > certain
>>>>>>> >> > esoteric cases.
>>>>>>> >>
>>>>>>> >> True, but ask anyone who manages a large cluster how
long it would
>>>>>>> >> take them to upgrade the jdk across their cluster and
validate all
>>>>>>> >> their applications and everything... binary compatibility
is a
>>>>>>> >> tiny
>>>>>>> >> drop in that bucket.
>>>>>>> >>
>>>>>>> >> --
>>>>>>> >> Marcelo
>>>>>>> >
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Marcelo
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>>
>>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Luciano Resende
>> http://twitter.com/lresende1975
>> http://lresende.blogspot.com/
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message