nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benson Margulies <bimargul...@gmail.com>
Subject Re: [Discuss] preparing for release 0.0.1
Date Thu, 08 Jan 2015 22:07:25 GMT
On Thu, Jan 8, 2015 at 4:53 PM, Joe Witt <joe.witt@gmail.com> wrote:

> Benson,
>
> Well I think we have all the groundwork laid as far as I know for the
> release plugin.  To be honest though (speaking for myself at least) I'm not
> sure what the steps would be that the release plugin would do.  I'd love it
> to be as you describe but am also concerned about how much we'd be bugging
> you to figure that out.  I looked at what other projects seem to do and it
> seemed shockingly manual.
>
> If you don't mind laying out the steps in more detail when you have time
> would love to learn from you on it.  I can also try toying around with it a
> bit more off-line.
>

I don't know what projects you're looking at, but it shouldn't look so
manual for any project that has a Maven top-level build.

Here's the basic workflow of the maven-release-plugin:

release:prepare:

1. edit poms to contain 'release version'.
2. make a commit.
3. make a tag.
4. edit poms to contain 'next snapshot'
5. make a commit.
6. push the whole business (unless you turn that off).

release:perform:

7. check out from tag in target/checkout (using extra git clone)
8. build
9. deploy results to target of deploymentManagement

Now comes the next trick.

Apache runs a copy of Sonatype Nexus, that serves as a 'staging
repository'.  So, when step 9 pushes things to the repository, it's pushing
them to a special 'purgatory' staging area. That area is available for
people to look at for testing, but does not propagate along to Maven
central.

Given all this, then we add one ingredient: an additional maven module that
uses the maven-assembly-plugin to build a tarball that satisfies the
requirements of an Apache source release. This is not the same thing as
what the maven-source-plugin does at all. Since it will have
<attach>true</attach>, the resulting tarball swims upstream to the staging
repo with everything else.

The vote email points people to that location, and supplies a sha1 for the
source ball. Voters download the source release from the staging repo, do
what they need to do, and vote. Three +1's later, and then you get to
publish.

Does this help? The only manual steps are really copying to svn to push to
the Apache dist area. CXF and all the Maven plugins are examples.




>
> Thanks
> Joe
>
> On Thu, Jan 8, 2015 at 4:49 PM, Benson Margulies <bimargulies@gmail.com>
> wrote:
>
> > On Thu, Jan 8, 2015 at 4:41 PM, Joe Witt <joe.witt@gmail.com> wrote:
> >
> > > Benson,
> > >
> > > Yes that would be much appreciated.
> > >
> > > Here is a rough gamplan of what I *think* will be needed to do the
> > release:
> > >
> > > - Branch from develop against a ticket for the overall release process
> > > - Update the version of all the artifacts and dependencies to
> > > 0.0.1-RC1-incubating or something like that
> > > - Create the tar.gz of the clean source tree.  Sign that and place both
> > the
> > > signed file and the tar.gz into my people's dir (if i have one of
> those)
> > > - Notify folks that this is available so they can pull it and attempt
> to
> > > build from that and validate the signature
> > > - Once folks agree it is good to merge that to master
> > > - Create a tar.gz of that and a signed file.  Call a vote within the
> > team.
> > > If that is good call a vote within IPMC.
> > > - IF NO - fix whatever is wrong.
> > > - If all good then do a maven deploy of the binary artifacts, source
> > > bundles, and javadocs
> > > - Upload the tar.gz of source and the signed file to our dist dir.  And
> > > upload convenience binary build tar.gz and zip with sigs to our dist
> > > folder.  Then create links to these from our downloads page.
> > > - Update JIRA that the release is closed.
> > > - Wash/Rinse/Repeat
> > >
> > > <note this all is with the understanding that we've done all the
> > necessary
> > > review of dependencies, licenses, properly documented them, have our
> > > ECCN/encryption stuff properly filed>
> > >
> >
> > Sure, except that I think we can make the maven-release-plugin do a lot
> of
> > the work.
> >
> > The idea is that you run the release plugin, and it delivers all these
> > goodies to repository.apache.org. Then you call the vote. If the vote
> > passes, you (a) push the promote button to push to Maven Central, and (b)
> > check the official source release tarball into the svn area for
> > distribution.
> >
> > I will take a whack at a PR for point (a).
> >
> >
> >
> >
> > >
> > > Might be missing a detail or two but does that sound roughly on the
> right
> > > rack?
> > >
> > > I had though the maven release plugin would be useful but looks like
> > we'll
> > > just be able to use the versions plugin to update them and can do the
> > other
> > > simple steps manually.
> > >
> > > Thanks
> > > Joe
> > >
> > > On Thu, Jan 8, 2015 at 3:46 PM, Benson Margulies <
> bimargulies@gmail.com>
> > > wrote:
> > >
> > > > Typically people set the maven assembly plugin for this and include
> it
> > in
> > > > the build. Would you like me to pitch this in?
> > > >
> > > >
> > > > On Thu, Jan 8, 2015 at 2:48 PM, Mike Drob <madrob@cloudera.com>
> wrote:
> > > >
> > > > > On Wed, Jan 7, 2015 at 9:22 PM, Joe Witt <joe.witt@gmail.com>
> wrote:
> > > > >
> > > > > > Josh
> > > > > >
> > > > > > Awesome.
> > > > > >
> > > > > > Honestly I believe we're good on the LICENSE, NOTICE, DISCLAIMER,
> > > > README.
> > > > > > All dependencies have been reviewed and checked for ASLv2
> > > > compatibility.
> > > > > >
> > > > > > Will submit the infra ticket now for the dist/ path for keys
> files
> > > and
> > > > > > such.
> > > > > >
> > > > > > My guess is that the release artifact should be a tarball of
all
> > > > source.
> > > > > > Could we literally just package up a clean source tree?  Anyone
> > else
> > > > have
> > > > > > views on this?
> > > > > >
> > > > >
> > > > > git archive -o [nifi-0.0.1-incubating.tar.gz] [release-tag] is a
> > > > perfectly
> > > > > reasonable thing to do.
> > > > >
> > > > > >
> > > > > > Ideally with this release we'd do it all properly including
maven
> > > > > > artifacts, sources/javadocs, and so on.  The Maven build does
> > already
> > > > > > operate now off a single command at the root to build everything
> > > > > > (build-order is gone) and inherits from the apache parent.
> > > > > >
> > > > > > Will need to incorporate RAT.
> > > > > >
> > > > > > Thanks for all that - definitely gives some stuff to work on
and
> > look
> > > > > into.
> > > > > >
> > > > > > Thanks
> > > > > > Joe
> > > > > >
> > > > > > On Wed, Jan 7, 2015 at 10:10 PM, Josh Elser <
> josh.elser@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Regardless of what you call it, writing down exactly what
was
> > done
> > > to
> > > > > > make
> > > > > > > a RC is extremely important early on (I know that I sure
can't
> > > > remember
> > > > > > > what I did last week, much less last release). I'm not
sure if
> > > > release
> > > > > > > guide is formally defined somewhere, but enough information
> that
> > > any
> > > > > > other
> > > > > > > committer can come by after "you get hit by a train" and
make a
> > > > release
> > > > > > is
> > > > > > > extremely important. Using the CMS and making a page on
the
> > website
> > > > for
> > > > > > > this is extremely easy to do, IMO.
> > > > > > >
> > > > > > > Another concern for how to actually make a release is the
type
> of
> > > > > > > packaging that is released. I not talking about the
> source/binary
> > > > > release
> > > > > > > here, literally "what is Apache NiFi 0.0.1-incubating"?
Is it a
> > > > > tarball?
> > > > > > > Are there jars in Maven? etc. A tarball is pretty easy
to make,
> > and
> > > > > > likely
> > > > > > > the closest to what the build-order.sh script already does
> (with
> > > > > Maven's
> > > > > > > help). I'm not sure if maven-released artifacts are quite
as
> > > > important
> > > > > at
> > > > > > > this point -- if it's not, punting on that will help. If/when
> you
> > > get
> > > > > to
> > > > > > > that point, look into the apache parent pom[1]. It is extremely
> > > well
> > > > > > > automated to use the existing infrastructure to do it all
for
> > you.
> > > > > > >
> > > > > > > Without looking at official documentation (which is me
being
> > lazy,
> > > > > > sorry),
> > > > > > > some other things that will need to be thought about:
> > > > > > >
> > > > > > > Start with the releases page [2]
> > > > > > >
> > > > > > > * You *must* include a source artifact. It cannot have
> binaries.
> > No
> > > > > Java
> > > > > > > class files, no jars. A user must be able to download the
> source
> > > > > artifact
> > > > > > > and build it all themselves. Artifacts which include binaries
> are
> > > for
> > > > > > > user-convenience only and can optionally be provided as
well.
> > > > > > >
> > > > > > > * LICENSE, NOTICE and README must all be included in the
top
> > level
> > > of
> > > > > the
> > > > > > > artifact. The NOTICE is the most painful and must include
any
> > > > required
> > > > > > > third-party notices. [3]
> > > > > > >
> > > > > > > * You will need to audit your dependencies to make sure
that
> they
> > > are
> > > > > > > ASLv2 compatible [4]
> > > > > > >
> > > > > > > * Releases must be cryptographically signed (PGP) [5].
Your
> > public
> > > > key
> > > > > > > should be published to known sites (e.g. pgp.mit.edu) and
must
> > be
> > > > > listed
> > > > > > > in NiFi's KEYS file [6] (which does not yet exist, probably
> needs
> > > an
> > > > > > infra
> > > > > > > ticket to create your dist/ directory?).
> > > > > > >
> > > > > > > * Verify that every source file contain the proper ASL
header.
> > The
> > > > > > > release-audit-tool[7] (`mvn apache-rat:check`) is a wonderful
> > tool
> > > > that
> > > > > > can
> > > > > > > (and should, IMO) be integrated into your build process
to
> > prevent
> > > > > people
> > > > > > > from accidentally committing new files between releases
that do
> > not
> > > > > have
> > > > > > > the correct header.
> > > > > > >
> > > > > > > And suddenly, this is really long :). Short answer: decide
what
> > the
> > > > > > > release should look like (just a tarball?), start vetting
your
> > > source
> > > > > > code
> > > > > > > for license headers, and looking into NiFi's dependencies
and
> > their
> > > > > > > licenses.
> > > > > > >
> > > > > > > [1] http://maven.apache.org/pom/asf/
> > > > > > > [2] http://www.apache.org/dev/release.html
> > > > > > > [3] http://www.apache.org/legal/src-headers.html
> > > > > > > [4] http://www.apache.org/legal/resolved.html#category-x
> > > > > > > [5] http://www.apache.org/dev/release-signing.html
> > > > > > > [6] http://www.apache.org/dist/incubator/nifi/KEYS
> > > > > > > [7] http://creadur.apache.org/rat/
> > > > > > >
> > > > > > >
> > > > > > > Tony Kurc wrote:
> > > > > > >
> > > > > > >> I read the guide Joe linked and a lot of the sticky
parts are
> > > marked
> > > > > > >> "TODO"
> > > > > > >> and it looks like a work in progress
> > > > > > >>
> > > > > > >> "Podlings can short circuit this process by starting
out with
> > > > written
> > > > > > >> release documentation. It is strongly recommended that
> Podlings
> > > > invest
> > > > > > >> time
> > > > > > >> looking at the best practices recommended in this document.
By
> > > > > selection
> > > > > > >> and modification, sections of this document can be
used to
> > quickly
> > > > and
> > > > > > >> easily bootstrap a release guide. "
> > > > > > >>
> > > > > > >> Is step one putting together a release guide?
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Wed, Jan 7, 2015 at 9:26 PM, Joe Witt<joe.witt@gmail.com>
> > > > wrote:
> > > > > > >>
> > > > > > >>  Hello
> > > > > > >>>
> > > > > > >>> Just wanted to stir this one up a bit.  Looks like
all
> tickets
> > > > pegged
> > > > > > as
> > > > > > >>> 0.0.1 are resolved (2 remain as of now but seem
likely to be
> > > > resolved
> > > > > > >>> shortly based on their comments).  So working through
the
> > release
> > > > > steps
> > > > > > >>> available on apache.org and via the link Brock
sent.
> > > > > > >>>
> > > > > > >>> Anyone interested in this part of the process or
who has
> advice
> > > to
> > > > > help
> > > > > > >>> us
> > > > > > >>> avoid landmines we're happy to hear it.
> > > > > > >>>
> > > > > > >>> Thanks
> > > > > > >>> Joe
> > > > > > >>>
> > > > > > >>> On Thu, Dec 18, 2014 at 1:31 PM, Joe Witt<joe.witt@gmail.com
> >
> > > > > wrote:
> > > > > > >>>
> > > > > > >>>  Thanks Brock this is very helpful.
> > > > > > >>>>
> > > > > > >>>> On Thu, Dec 18, 2014 at 1:27 PM, Brock Noland<
> > > brock@cloudera.com>
> > > > > > >>>>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>>> Hi,
> > > > > > >>>>>
> > > > > > >>>>> This is a decent guide which can be copied:
> > > > > > >>>>>
> > > > > > >>>>> http://mrunit.apache.org/pmc/how_to_release.html
> > > > > > >>>>>
> > > > > > >>>>> Brock
> > > > > > >>>>>
> > > > > > >>>>> On Thu, Dec 18, 2014 at 10:17 AM, Joe Witt<
> > joe.witt@gmail.com>
> > > > > > wrote:
> > > > > > >>>>>
> > > > > > >>>>>> Folks,
> > > > > > >>>>>>
> > > > > > >>>>>> Looking at the tickets that remain
which are presently
> tied
> > to
> > > > > 0.0.1
> > > > > > >>>>>>
> > > > > > >>>>> we're
> > > > > > >>>>>
> > > > > > >>>>>> probably 1-2 weeks out from this initial
release.  Can you
> > > > provide
> > > > > > >>>>>>
> > > > > > >>>>> some
> > > > > > >>>
> > > > > > >>>> pointers/references or pointers on how to get
this ball
> > rolling
> > > > and
> > > > > > >>>>>>
> > > > > > >>>>> any
> > > > > > >>>
> > > > > > >>>> rocks we must move before going down this path?
> > > > > > >>>>>>
> > > > > > >>>>>> http://incubator.apache.org/guides/releasemanagement.html
> > > > > > >>>>>>
> > > > > > >>>>>> That link seems really helpful but
has a lot of TODOs for
> > > areas
> > > > > > which
> > > > > > >>>>>>
> > > > > > >>>>> need
> > > > > > >>>>>
> > > > > > >>>>>> more explanation.
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> Thanks
> > > > > > >>>>>> Joe
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message