nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?
Date Thu, 16 May 2019 19:34:23 GMT
The last few items tagged for 0.4.0 have been merged so we should be
good to put out an RC for 0.4.0.

I'll start working on getting everything together and try to get
something out soon.

On Fri, May 10, 2019 at 9:00 AM Bryan Bende <bbende@gmail.com> wrote:
>
> I think we should be able to kick 0.4.0 very soon, just a couple of
> items to get in. Assuming those get reviewed and merged soon we could
> possibly kick out an RC early next week. I can plan to be RM unless
> anyone else is interested.
>
> On Thu, Mar 28, 2019 at 3:39 PM Bryan Bende <bbende@gmail.com> wrote:
> >
> > I think Kevin summarized it nicely. Auto-pulling the extensions with a
> > versioned flow is definitely the ultimate vision, but we need to take
> > steps to get there.
> >
> > The process of automatically bringing in the extensions is something
> > that would be implemented on the NiFi side and will require a bit of
> > thought and design around the user experience.
> >
> > Before we can do any of that, we first need somewhere to store and
> > retrieve versioned extensions, which is the work that is underway on
> > the NiFi Registry side.
> >
> > On Thu, Mar 28, 2019 at 3:29 PM Kevin Doran <kdoran@apache.org> wrote:
> > >
> > > Hi Ryan,
> > >
> > > Bryan can speak more to this than I can, but, yes, providing the
> > > experience you describe is the eventual goal! Aside from the NiFi
> > > Extension Registry support though (the initial bits of which is what
> > > Bryan is referring to in 0.4.0), getting the full, streamlined
> > > experience in NiFi will require changes to NiFi as well, for example
> > > in the logic to import a flow as it now also has to reach back to
> > > Registry to pull extensions referenced by the flow during import. So
> > > the changes to NiFi Registry to support hosting NiFi Extensions is
> > > only half of the puzzle. This is why Bryan also mentioned it might be
> > > worth holding back 0.4.0 (or releasing it, but marking the new
> > > extension stuff as experimental), as it is likely the current
> > > implementation and interfaces will need to change once the NiFi work
> > > begins.
> > >
> > > Cheers,
> > > Kevin
> > >
> > > On Thu, Mar 28, 2019 at 2:56 PM Ryan Hendrickson
> > > <ryan.andrew.hendrickson@gmail.com> wrote:
> > > >
> > > > We'd like to connect a NiFi to a Registry, Pull down a Process Group,
and
> > > > get any NARs required for that without having to manually install them
on
> > > > the NiFi.
> > > >
> > > > That's what I'm understanding the Extension part of the 0.4.0 release
to
> > > > be, am I on point, or way off?
> > > >
> > > > Thanks,
> > > > Ryan
> > > >
> > > > On Wed, Mar 27, 2019 at 4:17 PM Bryan Rosander <bryanrosander@gmail.com>
> > > > wrote:
> > > >
> > > > > Hey Pierre,
> > > > >
> > > > > You can version the metadata db in the root dir of the git repo with
event
> > > > > hooks.
> > > > >
> > > > > With an h2 jdbc url of
> > > > >
> > > > > 'jdbc:h2:$NIFI_REGISTRY_HOME/database/nifi-registry-primary;AUTOCOMMIT=OFF;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=TRUE'
> > > > >
> > > > > You can connect to the db in a separate process, we're using that
with a
> > > > > shell event hook to dump the db to sql script when mutations happen
(now
> > > > > tenant events too :) ), storing that in the git repo using
> > > > > org.h2.tools.Script, and committing it.  Then we can restore from
that sql
> > > > > script on startup before calling the stock startup script.
> > > > >
> > > > > Side benefit is you can see metadata state changes in the git history
and
> > > > > manually inspect the sql.
> > > > >
> > > > >
> > > > > On Wed, Mar 27, 2019 at 1:57 PM Pierre Villard <
> > > > > pierre.villard.fr@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Bryan,
> > > > > >
> > > > > > Thanks for the details. On my side, the main feature I (and
others I
> > > > > > suppose) am looking for is the ability to rebuild the metadata
DB from
> > > > > the
> > > > > > Git repo. But to be honest, this is not a blocker at all: I
can live
> > > > > with a
> > > > > > custom Docker image containing this feature.
> > > > > >
> > > > > > So, from my point of view, I don't have any issue with #2 but
maybe #1
> > > > > > would also give access to experimental features and have interesting
> > > > > > feedback from the community. But we would, indeed, need to make
it
> > > > > crystal
> > > > > > clear that the APIs could change in ulterior release(s).
> > > > > >
> > > > > > Pierre
> > > > > >
> > > > > >
> > > > > >
> > > > > > Le mer. 27 mars 2019 à 18:11, Bryan Bende <bbende@gmail.com>
a écrit :
> > > > > >
> > > > > > > Ryan,
> > > > > > >
> > > > > > > The short answer to your question is no, there isn't actually
anything
> > > > > > > new in the UI. The registry UI is setup in a generic way
to list
> > > > > > > "versioned items", so versioned extension bundles will
just
> > > > > > > automatically show up in the list with versioned flows.
> > > > > > >
> > > > > > > Let me put together a wiki page that outlines all of the
moving parts
> > > > > > > for the extension registry work and where we are currently
at. I'll
> > > > > > > send the link on this thread when I have something.
> > > > > > >
> > > > > > >
> > > > > > > For Pierre and others,
> > > > > > >
> > > > > > > I was thinking more about this discussion of releasing
0.4.0, and I
> > > > > > > think there are two of approaches we could take...
> > > > > > >
> > > > > > > 1) If there is a desire for the community to release 0.4.0
sooner
> > > > > > > rather than later, then we can mark all of the extension
bundle
> > > > > > > related APIs as experimental, which would then give us
the freedom to
> > > > > > > continue evolving them until we have the full extension
registry
> > > > > > > experience, and likely mark them as stable in 0.5.0, or
whichever
> > > > > > > future release we decide. The reason for this is that there
will
> > > > > > > likely be API changes needed while building out the NiFi
side of
> > > > > > > things, and I don't want us to get stuck in a spot where
we can't
> > > > > > > change some API since it has been released, and so far
the NiFi side
> > > > > > > of the work hasn't been started yet.
> > > > > > >
> > > > > > > 2) Wait until more of the extension registry work is finalized
and use
> > > > > > > that as the driver of when to release 0.4.0, with the obvious
downside
> > > > > > > that it may take a bit longer to get a release out which
then holds up
> > > > > > > anything else that is not related to extension registry.
> > > > > > >
> > > > > > > So I guess we just have to decide the driving factor for
releasing
> > > > > > > 0.4.0... If its the non-extension related features/fixes,
then #1 is
> > > > > > > probably the best approach. If its the extension related
stuff, then
> > > > > > > I'd vote for #2 since it isn't really ready yet.
> > > > > > >
> > > > > > > -Bryan
> > > > > > >
> > > > > > > On Wed, Mar 27, 2019 at 12:01 PM Ryan Hendrickson
> > > > > > > <ryan.andrew.hendrickson@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Bryan,
> > > > > > > >    Is there a new GUI part of the registry for the
extension registry
> > > > > > > work
> > > > > > > > in master?  I compiled 0.4.0-SNAPSHOT and didn't see
anything
> > > > > > > immediately.
> > > > > > > > You mentioned docs and sys admin stuff... Could you
point me in the
> > > > > > > > quick-start path if I wanted to try to kick-the-tires
a little on
> > > > > this?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Ryan
> > > > > > > >
> > > > > > > > On Tue, Mar 26, 2019 at 9:57 AM Bryan Bende <bbende@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Mike,
> > > > > > > > >
> > > > > > > > > All record-oriented components are extensions
and thus can make use
> > > > > > of
> > > > > > > > > versioned components (i.e. nothing related to
records is baked into
> > > > > > > > > the core framework of NiFi).
> > > > > > > > >
> > > > > > > > > The record API is essentially the module named
> > > > > > > > > nifi-record-serialization-services-api which
at runtime is included
> > > > > > in
> > > > > > > > > nifi-standard-services-api-nar. Any record oriented
processors you
> > > > > > > > > build are dependent on a specific version of
> > > > > > > > > nifi-standard-services-api-nar since that is
where the Java
> > > > > interface
> > > > > > > > > will be that your processor was compiled against.
> > > > > > > > >
> > > > > > > > > Right now, even without extension registry, you
could deploy
> > > > > multiple
> > > > > > > > > versions of nifi-standard-services-api-nar to
a NiFi instance, and
> > > > > > > > > then deploy multiple versions of your record
processing NAR that
> > > > > > > > > correspond to the versions of your API NAR.
> > > > > > > > >
> > > > > > > > > Going forward with extension registry, we probably
want to consider
> > > > > > > > > breaking up nifi-standard-services-api-nar into
smaller API NARs,
> > > > > as
> > > > > > > > > well as nifi-standard-bundle into smaller processor
bundles. For
> > > > > > > > > example, there could be a record API NAR and
a record processors
> > > > > NAR,
> > > > > > > > > that would both be split out of from the standard
NARs.
> > > > > > > > >
> > > > > > > > > -Bryan
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Mar 26, 2019 at 7:55 AM Mike Thomsen
<
> > > > > mikerthomsen@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Great news about the extension registry!
As we get closer to that
> > > > > > > being
> > > > > > > > > > ready, I'd like to add in some discussion
about versioning the
> > > > > > > record API
> > > > > > > > > > separately. A lot of the custom processors
we build now are
> > > > > Record
> > > > > > > > > > API-based, and it would be great to be able
to decouple them from
> > > > > > one
> > > > > > > > > > specific release of NiFi.
> > > > > > > > > >
> > > > > > > > > > On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende
<bbende@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Pierre,
> > > > > > > > > > >
> > > > > > > > > > > I think we are definitely close to
an 0.4.0 release. A major
> > > > > > chunk
> > > > > > > of
> > > > > > > > > > > extension registry work has already
landed in master, and I
> > > > > still
> > > > > > > have
> > > > > > > > > one
> > > > > > > > > > > other jira that I almost have ready
and wanted to include,
> > > > > > > NIFIREG-233
> > > > > > > > > for
> > > > > > > > > > > generating extensions docs. Plus we
probably need to make a few
> > > > > > > > > > > additions/updates to the admin guide.
> > > > > > > > > > >
> > > > > > > > > > > -Bryan
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Mar 25, 2019 at 1:33 PM Pierre
Villard <
> > > > > > > > > > > pierre.villard.fr@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > All,
> > > > > > > > > > > >
> > > > > > > > > > > > Some really nice features have
been included since the NiFi
> > > > > > > Registry
> > > > > > > > > > > 0.3.0
> > > > > > > > > > > > release and I'm wondering if it'd
be a good time to consider
> > > > > a
> > > > > > > 0.4.0
> > > > > > > > > > > > release.
> > > > > > > > > > > >
> > > > > > > > > > > > There is a JIRA tagged for 0.4.0
and there are 2 opened pull
> > > > > > > > > requests,
> > > > > > > > > > > plus
> > > > > > > > > > > > interesting discussions / JIRAS
on the mailing lists.
> > > > > > > > > > > >
> > > > > > > > > > > > Are there any reasons to hold
off on a 0.4.0 release?  Are
> > > > > > there
> > > > > > > > > > > particular
> > > > > > > > > > > > JIRAs that the community considers
necessary to have as part
> > > > > of
> > > > > > > the
> > > > > > > > > > > > release?
> > > > > > > > > > > >
> > > > > > > > > > > > If not, happy to give it a try
at performing RM duties!
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Pierre
> > > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Sent from Gmail Mobile
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >

Mime
View raw message