spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <>
Subject Re: time for Apache Spark 3.0?
Date Thu, 05 Apr 2018 12:45:26 GMT
On Wed, Apr 4, 2018 at 6:20 PM Reynold Xin <> wrote:

> The primary motivating factor IMO for a major version bump is to support
> Scala 2.12, which requires minor API breaking changes to Spark’s APIs.
> Similar to Spark 2.0, I think there are also opportunities for other
> changes that we know have been biting us for a long time but can’t be
> changed in feature releases (to be clear, I’m actually not sure they are
> all good ideas, but I’m writing them down as candidates for consideration):

IIRC from looking at this, it is possible to support 2.11 and 2.12
simultaneously. The cross-build already works now in 2.3.0. Barring some
big change needed to get 2.12 fully working -- and that may be the case --
it nearly works that way now.

Compiling vs 2.11 and 2.12 does however result in some APIs that differ in
byte code. However Scala itself isn't mutually compatible between 2.11 and
2.12 anyway; that's never been promised as compatible.

(Interesting question about what *Java* users should expect; they would see
a difference in 2.11 vs 2.12 Spark APIs, but that has always been true.)

I don't disagree with shooting for Spark 3.0, just saying I don't know if
2.12 support requires moving to 3.0. But, Spark 3.0 could consider dropping
2.11 support if needed to make supporting 2.12 less painful.

View raw message