spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <>
Subject Re: Straw poll: dropping support for things like Scala 2.10
Date Sat, 25 Feb 2017 19:10:20 GMT
I want to bring up the issue of Scala 2.10 support again, to see how people
feel about it. Key opinions from the previous responses, I think:

Cody: only drop 2.10 support when 2.12 support is added
Koert: we need all dependencies to support 2.12; Scala updates are pretty
transparent to IT/ops
Ofir: make sure to deprecate 2.10 in Spark 2.1
Reynold: let’s maybe remove support for Scala 2.10 and Java 7 in Spark 2.2
Matei: let’s not remove things unless they’re burdensome for the project;
some people are still on old environments that their IT can’t easily update

Scala 2.10 support was deprecated in 2.1, and we did remove Java 7 support
for 2.2. tracks the work
to support 2.12, and there is progress, especially in dependencies
supporting 2.12.

It looks like 2.12 support may even entail a breaking change as documented
in and will mean dropping
Kafka 0.8, for example. In any event it’s going to take some surgery and a
few hacks to make one code base work across 2.11 and 2.12. I don’t see this
happening for Spark 2.2.0 because there are just a few weeks left.

Supporting three versions at once is probably infeasible, so dropping 2.10
should precede 2.12 support. Right now, I would like to make progress
towards changes that 2.12 will require but that 2.11/2.10 can support. For
example, we have to update scalatest, breeze, chill, etc. and can do that
before 2.12 is enabled. However I’m finding making those changes tricky or
maybe impossible in one case while 2.10 is still supported.

For 2.2.0, I’m wondering if it makes sense to go ahead and drop 2.10
support, and even get in additional prep work for 2.12, into the 2.2.0
release. The move to support 2.12 in 2.3.0 would then be a smaller change.
It isn’t strictly necessary. We could delay all of that until after 2.2.0
and get it all done between 2.2.0 and 2.3.0. But I wonder if 2.10 is legacy
enough at this stage to drop for Spark 2.2.0?

I don’t feel strongly about it but there are some reasonable arguments for
dropping it:

- 2.10 doesn’t technically support Java 8, though we do have it working
still even after requiring Java 8
- Safe to say virtually all common _2.10 libraries has a _2.11 counterpart
at this point?
- 2.10.x was “EOL” in September 2015 with the final 2.10.6 release
- For a vendor viewpoint: CDH only supports Scala 2.11 with Spark 2.x

Before I open a JIRA, just soliciting opinions.

On Tue, Oct 25, 2016 at 4:36 PM Sean Owen <> wrote:

> I'd like to gauge where people stand on the issue of dropping support for
> a few things that were considered for 2.0.
> First: Scala 2.10. We've seen a number of build breakages this week
> because the PR builder only tests 2.11. No big deal at this stage, but, it
> did cause me to wonder whether it's time to plan to drop 2.10 support,
> especially with 2.12 coming soon.
> Next, Java 7. It's reasonably old and out of public updates at this stage.
> It's not that painful to keep supporting, to be honest. It would simplify
> some bits of code, some scripts, some testing.
> Hadoop versions: I think the the general argument is that most anyone
> would be using, at the least, 2.6, and it would simplify some code that has
> to reflect to use not-even-that-new APIs. It would remove some moderate
> complexity in the build.
> "When" is a tricky question. Although it's a little aggressive for minor
> releases, I think these will all happen before 3.x regardless. 2.1.0 is not
> out of the question, though coming soon. What about ... 2.2.0?
> Although I tend to favor dropping support, I'm mostly asking for current
> opinions.

View raw message