spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <>
Subject Re: Are we moving too fast or too far on 0.8.1-SNAPSHOT?
Date Mon, 28 Oct 2013 23:06:21 GMT
So far we've maintained backwards compatibility for point releases in everything except internal
or alpha APIs. One place where it changed, for example, is when Twitter removed its username/password
authentication method, which required us to change the Spark Streaming API (but that was actually
alpha). Which APIs did you see broken?

The problem is likely that Shark calls into on some internal APIs, which we should fix in
the future to put it on more equal footing with other clients. But you can expect that Shark
releases from our team will work with the corresponding Spark release, and user programs written
for Spark 0.8.0 will definitely work for 0.8.1.


On Oct 28, 2013, at 3:50 PM, Jey Kottalam <> wrote:

> I agree that we should strive to maintain full backward compatibility
> between patch releases (i.e. incrementing the "z" in "version x.y.z").
> On Mon, Oct 28, 2013 at 3:22 PM, Mark Hamstra <> wrote:
>> Or more to the point: What is our commitment to backward compatibility in
>> point releases?
>> Many Java developers will come to a library or platform versioned as x.y.z
>> with the expectation that if their own code worked well using x.y.(z-1) as
>> a dependency, then moving up to x.y.z will be painless and trivial.  That
>> is not looking like it will be the case for Spark 0.8.0 and 0.8.1.
>> We only need to look at Shark as an example of code built with a dependency
>> on Spark to see the problem.  Shark 0.8.0 works with Spark 0.8.0.  Shark
>> 0.8.0 does not build with Spark 0.8.1-SNAPSHOT.  Presumably that lack of
>> backwards compatibility will continue into the eventual release of Spark
>> 0.8.1, and that makes life hard on developers using Spark and Shark.  For
>> example, a developer using the released version of Shark but wanting to
>> pick up the bug fixes in Spark doesn't have a good option anymore since
>> 0.8.1-SNAPSHOT (or the eventual 0.8.1 release) doesn't work, and moving to
>> the wild and woolly development on the master branches of Spark and Shark
>> is not a good idea for someone trying to develop production code.  In other
>> words, all of the bug fixes in Spark 0.8.1 are not accessible to this
>> developer until such time as there are available 0.8.1-compatible versions
>> of Shark and anything else built on Spark that this developer is using.
>> The only other option is trying to cherry-pick commits from, e.g., Shark
>> 0.9.0-SNAPSHOT into Shark 0.8.0 until Shark 0.8.0 has been brought up to a
>> point where it works with Spark 0.8.1.  But an application developer
>> shouldn't need to do that just to get the bug fixes in Spark 0.8.1, and it
>> is not immediately obvious just which Shark commits are necessary and
>> sufficient to produce a correct, Spark-0.8.1-compatible version of Shark
>> (indeed, there is no guarantee that such a thing is even possible.)  Right
>> now, I believe that 67626ae3eb6a23efc504edf5aedc417197f072cf,
>> 488930f5187264d094810f06f33b5b5a2fde230a and
>> bae19222b3b221946ff870e0cee4dba0371dea04 are necessary to get Shark to work
>> with Spark 0.8.1-SNAPSHOT, but that those commits are not sufficient (Shark
>> builds against Spark 0.8.1-SNAPSHOT with those cherry-picks, but I'm still
>> seeing runtime errors.)
>> In short, this is not a good situation, and we probably need a real 0.8
>> maintenance branch that maintains backward compatibility with 0.8.0,
>> because (at least to me) the current branch-0.8 of Spark looks more like
>> another active development branch (in addition to the master and scala-2.10
>> branches) than it does a maintenance branch.

View raw message