spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <>
Subject Re: Spark 2.0.0-preview artifacts still not available in Maven
Date Wed, 01 Jun 2016 22:58:09 GMT
Hi Sean,

(writing this email with my Apache hat on only and not Databricks hat)

The preview release is available here: (there is an entire section
dedicated to it and also there is a news link to it on the right).

Again, I think this is a good opportunity to define what a release should
contain. Based on

"In addition to the distribution directory, project that use Maven or a
related build tool sometimes place their releases on
beside some convenience binaries. The distribution directory is required,
while the repository system is an optional convenience."

So I'm reading it as that maven publication is not necessary. My
understanding is that the general community (beyond who follows the dev
list) should understand that preview is not a stable release, and we as the
PMC should set expectations accordingly. Developers that can test the
preview releases tend to be more savvy and are comfortable on the bleeding
edge. It is actually fairly easy for them to add a maven repo. Now reading
the page I realized no where on the page did we mention the temporary maven
repo. I will fix that.

I think it'd be pretty bad if preview releases in anyway become "default
version", because they are unstable and contain a lot of blocker bugs.

So my concrete proposal is:

1. Separate (officially voted) releases into normal and preview.

2. On the download page, have two sections. One listing the normal
releases, and the other listing preview releases.

3. Everywhere we mention preview releases, include the proper disclaimer
e.g. "This preview is not a stable release in terms of either API or
functionality, but it is meant to give the community early access to try
the code that will become Spark 2.0."

4. Publish normal releases to maven central, and preview releases only to
the staging maven repo. But of course we should include the temporary maven
repo for preview releases on the download page.

On Wed, Jun 1, 2016 at 3:10 PM, Sean Owen <> wrote:

> I'll be more specific about the issue that I think trumps all this,
> which I realize maybe not everyone was aware of.
> There was a long and contentious discussion on the PMC about, among
> other things, advertising a "Spark 2.0 preview" from Databricks, such
> as at
> That post has already been updated/fixed from an earlier version, but
> part of the resolution was to make a full "2.0.0 preview" release in
> order to continue to be able to advertise it as such. Without it, I
> believe the PMC's conclusion remains that this blog post / product
> announcement is not allowed by ASF policy. Hence, either the product
> announcements need to be taken down and a bunch of wording changed in
> the Databricks product, or, this needs to be a normal release.
> Obviously, it seems far easier to just finish the release per usual. I
> actually didn't realize this had not been offered for download at
> either. It needs to be
> accessible there too.
> We can get back in the weeds about what a "preview" release means,
> but, normal voted releases can and even should be alpha/beta
> ( The culture is, in theory, to
> release early and often. I don't buy an argument that it's too old, at
> 2 weeks, when the alternative is having nothing at all to test
> against.
> On Wed, Jun 1, 2016 at 5:02 PM, Michael Armbrust <>
> wrote:
> >> I'd think we want less effort, not more, to let people test it? for
> >> example, right now I can't easily try my product build against
> >> 2.0.0-preview.
> >
> >
> > I don't feel super strongly one way or the other, so if we need to
> publish
> > it permanently we can.
> >
> > However, either way you can still test against this release.  You just
> need
> > to add a resolver as well (which is how I have always tested packages
> > against RCs).  One concern with making it permeant is this preview
> release
> > is already fairly far behind branch-2.0, so many of the issues that
> people
> > might report have already been fixed and that might continue even after
> the
> > release is made.  I'd rather be able to force upgrades eventually when we
> > vote on the final 2.0 release.
> >

View raw message