nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Taft <a...@adamtaft.com>
Subject Re: [Discuss] preparing for release 0.0.1
Date Thu, 08 Jan 2015 22:36:32 GMT
Benson,

Quite a bit of NIFI's artifacts are more like application bundles of other
common libraries.  Most of the NIFI nars aren't really useful for
application developers to bind to in their own code.  These nars are more
part of the application and less useful as a shared common library.

One of my concerns was accidentally releasing unneeded nars into maven
central.  It sounds like the "purgatory" area solves this?  Can you speak
to this a little more?

There's been discussion on whether we care that application nars escape
into maven central - I'm not trying to weigh in on that topic (though, I
think you can read my thoughts between the lines).  I'm just asking for
more clarity on the options when using the maven release plugin.  Purgatory
might very well allow us to have our cake and eat it too.

Thanks,

Adam


On Thu, Jan 8, 2015 at 5:07 PM, Benson Margulies <bimargulies@gmail.com>
wrote:

> On Thu, Jan 8, 2015 at 4:53 PM, Joe Witt <joe.witt@gmail.com> wrote:
>
> > Benson,
> >
> > Well I think we have all the groundwork laid as far as I know for the
> > release plugin.  To be honest though (speaking for myself at least) I'm
> not
> > sure what the steps would be that the release plugin would do.  I'd love
> it
> > to be as you describe but am also concerned about how much we'd be
> bugging
> > you to figure that out.  I looked at what other projects seem to do and
> it
> > seemed shockingly manual.
> >
> > If you don't mind laying out the steps in more detail when you have time
> > would love to learn from you on it.  I can also try toying around with
> it a
> > bit more off-line.
> >
>
> I don't know what projects you're looking at, but it shouldn't look so
> manual for any project that has a Maven top-level build.
>
> Here's the basic workflow of the maven-release-plugin:
>
> release:prepare:
>
> 1. edit poms to contain 'release version'.
> 2. make a commit.
> 3. make a tag.
> 4. edit poms to contain 'next snapshot'
> 5. make a commit.
> 6. push the whole business (unless you turn that off).
>
> release:perform:
>
> 7. check out from tag in target/checkout (using extra git clone)
> 8. build
> 9. deploy results to target of deploymentManagement
>
> Now comes the next trick.
>
> Apache runs a copy of Sonatype Nexus, that serves as a 'staging
> repository'.  So, when step 9 pushes things to the repository, it's pushing
> them to a special 'purgatory' staging area. That area is available for
> people to look at for testing, but does not propagate along to Maven
> central.
>
> Given all this, then we add one ingredient: an additional maven module that
> uses the maven-assembly-plugin to build a tarball that satisfies the
> requirements of an Apache source release. This is not the same thing as
> what the maven-source-plugin does at all. Since it will have
> <attach>true</attach>, the resulting tarball swims upstream to the staging
> repo with everything else.
>
> The vote email points people to that location, and supplies a sha1 for the
> source ball. Voters download the source release from the staging repo, do
> what they need to do, and vote. Three +1's later, and then you get to
> publish.
>
> Does this help? The only manual steps are really copying to svn to push to
> the Apache dist area. CXF and all the Maven plugins are examples.
>
>
>
>
> >
> > Thanks
> > Joe
> >
> > On Thu, Jan 8, 2015 at 4:49 PM, Benson Margulies <bimargulies@gmail.com>
> > wrote:
> >
> > > On Thu, Jan 8, 2015 at 4:41 PM, Joe Witt <joe.witt@gmail.com> wrote:
> > >
> > > > Benson,
> > > >
> > > > Yes that would be much appreciated.
> > > >
> > > > Here is a rough gamplan of what I *think* will be needed to do the
> > > release:
> > > >
> > > > - Branch from develop against a ticket for the overall release
> process
> > > > - Update the version of all the artifacts and dependencies to
> > > > 0.0.1-RC1-incubating or something like that
> > > > - Create the tar.gz of the clean source tree.  Sign that and place
> both
> > > the
> > > > signed file and the tar.gz into my people's dir (if i have one of
> > those)
> > > > - Notify folks that this is available so they can pull it and attempt
> > to
> > > > build from that and validate the signature
> > > > - Once folks agree it is good to merge that to master
> > > > - Create a tar.gz of that and a signed file.  Call a vote within the
> > > team.
> > > > If that is good call a vote within IPMC.
> > > > - IF NO - fix whatever is wrong.
> > > > - If all good then do a maven deploy of the binary artifacts, source
> > > > bundles, and javadocs
> > > > - Upload the tar.gz of source and the signed file to our dist dir.
> And
> > > > upload convenience binary build tar.gz and zip with sigs to our dist
> > > > folder.  Then create links to these from our downloads page.
> > > > - Update JIRA that the release is closed.
> > > > - Wash/Rinse/Repeat
> > > >
> > > > <note this all is with the understanding that we've done all the
> > > necessary
> > > > review of dependencies, licenses, properly documented them, have our
> > > > ECCN/encryption stuff properly filed>
> > > >
> > >
> > > Sure, except that I think we can make the maven-release-plugin do a lot
> > of
> > > the work.
> > >
> > > The idea is that you run the release plugin, and it delivers all these
> > > goodies to repository.apache.org. Then you call the vote. If the vote
> > > passes, you (a) push the promote button to push to Maven Central, and
> (b)
> > > check the official source release tarball into the svn area for
> > > distribution.
> > >
> > > I will take a whack at a PR for point (a).
> > >
> > >
> > >
> > >
> > > >
> > > > Might be missing a detail or two but does that sound roughly on the
> > right
> > > > rack?
> > > >
> > > > I had though the maven release plugin would be useful but looks like
> > > we'll
> > > > just be able to use the versions plugin to update them and can do the
> > > other
> > > > simple steps manually.
> > > >
> > > > Thanks
> > > > Joe
> > > >
> > > > On Thu, Jan 8, 2015 at 3:46 PM, Benson Margulies <
> > bimargulies@gmail.com>
> > > > wrote:
> > > >
> > > > > Typically people set the maven assembly plugin for this and include
> > it
> > > in
> > > > > the build. Would you like me to pitch this in?
> > > > >
> > > > >
> > > > > On Thu, Jan 8, 2015 at 2:48 PM, Mike Drob <madrob@cloudera.com>
> > wrote:
> > > > >
> > > > > > On Wed, Jan 7, 2015 at 9:22 PM, Joe Witt <joe.witt@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Josh
> > > > > > >
> > > > > > > Awesome.
> > > > > > >
> > > > > > > Honestly I believe we're good on the LICENSE, NOTICE,
> DISCLAIMER,
> > > > > README.
> > > > > > > All dependencies have been reviewed and checked for ASLv2
> > > > > compatibility.
> > > > > > >
> > > > > > > Will submit the infra ticket now for the dist/ path for
keys
> > files
> > > > and
> > > > > > > such.
> > > > > > >
> > > > > > > My guess is that the release artifact should be a tarball
of
> all
> > > > > source.
> > > > > > > Could we literally just package up a clean source tree?
 Anyone
> > > else
> > > > > have
> > > > > > > views on this?
> > > > > > >
> > > > > >
> > > > > > git archive -o [nifi-0.0.1-incubating.tar.gz] [release-tag]
is a
> > > > > perfectly
> > > > > > reasonable thing to do.
> > > > > >
> > > > > > >
> > > > > > > Ideally with this release we'd do it all properly including
> maven
> > > > > > > artifacts, sources/javadocs, and so on.  The Maven build
does
> > > already
> > > > > > > operate now off a single command at the root to build
> everything
> > > > > > > (build-order is gone) and inherits from the apache parent.
> > > > > > >
> > > > > > > Will need to incorporate RAT.
> > > > > > >
> > > > > > > Thanks for all that - definitely gives some stuff to work
on
> and
> > > look
> > > > > > into.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Joe
> > > > > > >
> > > > > > > On Wed, Jan 7, 2015 at 10:10 PM, Josh Elser <
> > josh.elser@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Regardless of what you call it, writing down exactly
what was
> > > done
> > > > to
> > > > > > > make
> > > > > > > > a RC is extremely important early on (I know that
I sure
> can't
> > > > > remember
> > > > > > > > what I did last week, much less last release). I'm
not sure
> if
> > > > > release
> > > > > > > > guide is formally defined somewhere, but enough information
> > that
> > > > any
> > > > > > > other
> > > > > > > > committer can come by after "you get hit by a train"
and
> make a
> > > > > release
> > > > > > > is
> > > > > > > > extremely important. Using the CMS and making a page
on the
> > > website
> > > > > for
> > > > > > > > this is extremely easy to do, IMO.
> > > > > > > >
> > > > > > > > Another concern for how to actually make a release
is the
> type
> > of
> > > > > > > > packaging that is released. I not talking about the
> > source/binary
> > > > > > release
> > > > > > > > here, literally "what is Apache NiFi 0.0.1-incubating"?
Is
> it a
> > > > > > tarball?
> > > > > > > > Are there jars in Maven? etc. A tarball is pretty
easy to
> make,
> > > and
> > > > > > > likely
> > > > > > > > the closest to what the build-order.sh script already
does
> > (with
> > > > > > Maven's
> > > > > > > > help). I'm not sure if maven-released artifacts are
quite as
> > > > > important
> > > > > > at
> > > > > > > > this point -- if it's not, punting on that will help.
If/when
> > you
> > > > get
> > > > > > to
> > > > > > > > that point, look into the apache parent pom[1]. It
is
> extremely
> > > > well
> > > > > > > > automated to use the existing infrastructure to do
it all for
> > > you.
> > > > > > > >
> > > > > > > > Without looking at official documentation (which is
me being
> > > lazy,
> > > > > > > sorry),
> > > > > > > > some other things that will need to be thought about:
> > > > > > > >
> > > > > > > > Start with the releases page [2]
> > > > > > > >
> > > > > > > > * You *must* include a source artifact. It cannot
have
> > binaries.
> > > No
> > > > > > Java
> > > > > > > > class files, no jars. A user must be able to download
the
> > source
> > > > > > artifact
> > > > > > > > and build it all themselves. Artifacts which include
binaries
> > are
> > > > for
> > > > > > > > user-convenience only and can optionally be provided
as well.
> > > > > > > >
> > > > > > > > * LICENSE, NOTICE and README must all be included
in the top
> > > level
> > > > of
> > > > > > the
> > > > > > > > artifact. The NOTICE is the most painful and must
include any
> > > > > required
> > > > > > > > third-party notices. [3]
> > > > > > > >
> > > > > > > > * You will need to audit your dependencies to make
sure that
> > they
> > > > are
> > > > > > > > ASLv2 compatible [4]
> > > > > > > >
> > > > > > > > * Releases must be cryptographically signed (PGP)
[5]. Your
> > > public
> > > > > key
> > > > > > > > should be published to known sites (e.g. pgp.mit.edu)
and
> must
> > > be
> > > > > > listed
> > > > > > > > in NiFi's KEYS file [6] (which does not yet exist,
probably
> > needs
> > > > an
> > > > > > > infra
> > > > > > > > ticket to create your dist/ directory?).
> > > > > > > >
> > > > > > > > * Verify that every source file contain the proper
ASL
> header.
> > > The
> > > > > > > > release-audit-tool[7] (`mvn apache-rat:check`) is
a wonderful
> > > tool
> > > > > that
> > > > > > > can
> > > > > > > > (and should, IMO) be integrated into your build process
to
> > > prevent
> > > > > > people
> > > > > > > > from accidentally committing new files between releases
that
> do
> > > not
> > > > > > have
> > > > > > > > the correct header.
> > > > > > > >
> > > > > > > > And suddenly, this is really long :). Short answer:
decide
> what
> > > the
> > > > > > > > release should look like (just a tarball?), start
vetting
> your
> > > > source
> > > > > > > code
> > > > > > > > for license headers, and looking into NiFi's dependencies
and
> > > their
> > > > > > > > licenses.
> > > > > > > >
> > > > > > > > [1] http://maven.apache.org/pom/asf/
> > > > > > > > [2] http://www.apache.org/dev/release.html
> > > > > > > > [3] http://www.apache.org/legal/src-headers.html
> > > > > > > > [4] http://www.apache.org/legal/resolved.html#category-x
> > > > > > > > [5] http://www.apache.org/dev/release-signing.html
> > > > > > > > [6] http://www.apache.org/dist/incubator/nifi/KEYS
> > > > > > > > [7] http://creadur.apache.org/rat/
> > > > > > > >
> > > > > > > >
> > > > > > > > Tony Kurc wrote:
> > > > > > > >
> > > > > > > >> I read the guide Joe linked and a lot of the sticky
parts
> are
> > > > marked
> > > > > > > >> "TODO"
> > > > > > > >> and it looks like a work in progress
> > > > > > > >>
> > > > > > > >> "Podlings can short circuit this process by starting
out
> with
> > > > > written
> > > > > > > >> release documentation. It is strongly recommended
that
> > Podlings
> > > > > invest
> > > > > > > >> time
> > > > > > > >> looking at the best practices recommended in this
document.
> By
> > > > > > selection
> > > > > > > >> and modification, sections of this document can
be used to
> > > quickly
> > > > > and
> > > > > > > >> easily bootstrap a release guide. "
> > > > > > > >>
> > > > > > > >> Is step one putting together a release guide?
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Wed, Jan 7, 2015 at 9:26 PM, Joe Witt<joe.witt@gmail.com
> >
> > > > > wrote:
> > > > > > > >>
> > > > > > > >>  Hello
> > > > > > > >>>
> > > > > > > >>> Just wanted to stir this one up a bit.  Looks
like all
> > tickets
> > > > > pegged
> > > > > > > as
> > > > > > > >>> 0.0.1 are resolved (2 remain as of now but
seem likely to
> be
> > > > > resolved
> > > > > > > >>> shortly based on their comments).  So working
through the
> > > release
> > > > > > steps
> > > > > > > >>> available on apache.org and via the link Brock
sent.
> > > > > > > >>>
> > > > > > > >>> Anyone interested in this part of the process
or who has
> > advice
> > > > to
> > > > > > help
> > > > > > > >>> us
> > > > > > > >>> avoid landmines we're happy to hear it.
> > > > > > > >>>
> > > > > > > >>> Thanks
> > > > > > > >>> Joe
> > > > > > > >>>
> > > > > > > >>> On Thu, Dec 18, 2014 at 1:31 PM, Joe Witt<
> joe.witt@gmail.com
> > >
> > > > > > wrote:
> > > > > > > >>>
> > > > > > > >>>  Thanks Brock this is very helpful.
> > > > > > > >>>>
> > > > > > > >>>> On Thu, Dec 18, 2014 at 1:27 PM, Brock
Noland<
> > > > brock@cloudera.com>
> > > > > > > >>>>
> > > > > > > >>> wrote:
> > > > > > > >>>
> > > > > > > >>>> Hi,
> > > > > > > >>>>>
> > > > > > > >>>>> This is a decent guide which can be
copied:
> > > > > > > >>>>>
> > > > > > > >>>>> http://mrunit.apache.org/pmc/how_to_release.html
> > > > > > > >>>>>
> > > > > > > >>>>> Brock
> > > > > > > >>>>>
> > > > > > > >>>>> On Thu, Dec 18, 2014 at 10:17 AM,
Joe Witt<
> > > joe.witt@gmail.com>
> > > > > > > wrote:
> > > > > > > >>>>>
> > > > > > > >>>>>> Folks,
> > > > > > > >>>>>>
> > > > > > > >>>>>> Looking at the tickets that remain
which are presently
> > tied
> > > to
> > > > > > 0.0.1
> > > > > > > >>>>>>
> > > > > > > >>>>> we're
> > > > > > > >>>>>
> > > > > > > >>>>>> probably 1-2 weeks out from this
initial release.  Can
> you
> > > > > provide
> > > > > > > >>>>>>
> > > > > > > >>>>> some
> > > > > > > >>>
> > > > > > > >>>> pointers/references or pointers on how
to get this ball
> > > rolling
> > > > > and
> > > > > > > >>>>>>
> > > > > > > >>>>> any
> > > > > > > >>>
> > > > > > > >>>> rocks we must move before going down this
path?
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> http://incubator.apache.org/guides/releasemanagement.html
> > > > > > > >>>>>>
> > > > > > > >>>>>> That link seems really helpful
but has a lot of TODOs
> for
> > > > areas
> > > > > > > which
> > > > > > > >>>>>>
> > > > > > > >>>>> need
> > > > > > > >>>>>
> > > > > > > >>>>>> more explanation.
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> Thanks
> > > > > > > >>>>>> Joe
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message