spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <>
Subject Re: Are we moving too fast or too far on 0.8.1-SNAPSHOT?
Date Mon, 28 Oct 2013 23:02:20 GMT
Shark is not a great example in general because it uses semi-private
internal interfaces that are not guaranteed to be compatible within
minor releases.

Spark's public, documented API has always (AFAIK) maintained
compatibility within minor versions. In fact, we've been diligent to
maintain compatibility with major versions as well and there have only
been very minute changes in that API.

Over time it would be good for Shark to migrate to using higher API's
(and we may need to build these).

But my point is that the public API has maintained compatibility
consistent with the norms discussed here.

- Patrick

On Mon, Oct 28, 2013 at 3:50 PM, Jey Kottalam <> wrote:
> I agree that we should strive to maintain full backward compatibility
> between patch releases (i.e. incrementing the "z" in "version x.y.z").
> On Mon, Oct 28, 2013 at 3:22 PM, Mark Hamstra <> wrote:
>> Or more to the point: What is our commitment to backward compatibility in
>> point releases?
>> Many Java developers will come to a library or platform versioned as x.y.z
>> with the expectation that if their own code worked well using x.y.(z-1) as
>> a dependency, then moving up to x.y.z will be painless and trivial.  That
>> is not looking like it will be the case for Spark 0.8.0 and 0.8.1.
>> We only need to look at Shark as an example of code built with a dependency
>> on Spark to see the problem.  Shark 0.8.0 works with Spark 0.8.0.  Shark
>> 0.8.0 does not build with Spark 0.8.1-SNAPSHOT.  Presumably that lack of
>> backwards compatibility will continue into the eventual release of Spark
>> 0.8.1, and that makes life hard on developers using Spark and Shark.  For
>> example, a developer using the released version of Shark but wanting to
>> pick up the bug fixes in Spark doesn't have a good option anymore since
>> 0.8.1-SNAPSHOT (or the eventual 0.8.1 release) doesn't work, and moving to
>> the wild and woolly development on the master branches of Spark and Shark
>> is not a good idea for someone trying to develop production code.  In other
>> words, all of the bug fixes in Spark 0.8.1 are not accessible to this
>> developer until such time as there are available 0.8.1-compatible versions
>> of Shark and anything else built on Spark that this developer is using.
>> The only other option is trying to cherry-pick commits from, e.g., Shark
>> 0.9.0-SNAPSHOT into Shark 0.8.0 until Shark 0.8.0 has been brought up to a
>> point where it works with Spark 0.8.1.  But an application developer
>> shouldn't need to do that just to get the bug fixes in Spark 0.8.1, and it
>> is not immediately obvious just which Shark commits are necessary and
>> sufficient to produce a correct, Spark-0.8.1-compatible version of Shark
>> (indeed, there is no guarantee that such a thing is even possible.)  Right
>> now, I believe that 67626ae3eb6a23efc504edf5aedc417197f072cf,
>> 488930f5187264d094810f06f33b5b5a2fde230a and
>> bae19222b3b221946ff870e0cee4dba0371dea04 are necessary to get Shark to work
>> with Spark 0.8.1-SNAPSHOT, but that those commits are not sufficient (Shark
>> builds against Spark 0.8.1-SNAPSHOT with those cherry-picks, but I'm still
>> seeing runtime errors.)
>> In short, this is not a good situation, and we probably need a real 0.8
>> maintenance branch that maintains backward compatibility with 0.8.0,
>> because (at least to me) the current branch-0.8 of Spark looks more like
>> another active development branch (in addition to the master and scala-2.10
>> branches) than it does a maintenance branch.

View raw message