nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?
Date Fri, 10 May 2019 13:00:28 GMT
I think we should be able to kick 0.4.0 very soon, just a couple of
items to get in. Assuming those get reviewed and merged soon we could
possibly kick out an RC early next week. I can plan to be RM unless
anyone else is interested.

On Thu, Mar 28, 2019 at 3:39 PM Bryan Bende <bbende@gmail.com> wrote:
>
> I think Kevin summarized it nicely. Auto-pulling the extensions with a
> versioned flow is definitely the ultimate vision, but we need to take
> steps to get there.
>
> The process of automatically bringing in the extensions is something
> that would be implemented on the NiFi side and will require a bit of
> thought and design around the user experience.
>
> Before we can do any of that, we first need somewhere to store and
> retrieve versioned extensions, which is the work that is underway on
> the NiFi Registry side.
>
> On Thu, Mar 28, 2019 at 3:29 PM Kevin Doran <kdoran@apache.org> wrote:
> >
> > Hi Ryan,
> >
> > Bryan can speak more to this than I can, but, yes, providing the
> > experience you describe is the eventual goal! Aside from the NiFi
> > Extension Registry support though (the initial bits of which is what
> > Bryan is referring to in 0.4.0), getting the full, streamlined
> > experience in NiFi will require changes to NiFi as well, for example
> > in the logic to import a flow as it now also has to reach back to
> > Registry to pull extensions referenced by the flow during import. So
> > the changes to NiFi Registry to support hosting NiFi Extensions is
> > only half of the puzzle. This is why Bryan also mentioned it might be
> > worth holding back 0.4.0 (or releasing it, but marking the new
> > extension stuff as experimental), as it is likely the current
> > implementation and interfaces will need to change once the NiFi work
> > begins.
> >
> > Cheers,
> > Kevin
> >
> > On Thu, Mar 28, 2019 at 2:56 PM Ryan Hendrickson
> > <ryan.andrew.hendrickson@gmail.com> wrote:
> > >
> > > We'd like to connect a NiFi to a Registry, Pull down a Process Group, and
> > > get any NARs required for that without having to manually install them on
> > > the NiFi.
> > >
> > > That's what I'm understanding the Extension part of the 0.4.0 release to
> > > be, am I on point, or way off?
> > >
> > > Thanks,
> > > Ryan
> > >
> > > On Wed, Mar 27, 2019 at 4:17 PM Bryan Rosander <bryanrosander@gmail.com>
> > > wrote:
> > >
> > > > Hey Pierre,
> > > >
> > > > You can version the metadata db in the root dir of the git repo with event
> > > > hooks.
> > > >
> > > > With an h2 jdbc url of
> > > >
> > > > 'jdbc:h2:$NIFI_REGISTRY_HOME/database/nifi-registry-primary;AUTOCOMMIT=OFF;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=TRUE'
> > > >
> > > > You can connect to the db in a separate process, we're using that with
a
> > > > shell event hook to dump the db to sql script when mutations happen (now
> > > > tenant events too :) ), storing that in the git repo using
> > > > org.h2.tools.Script, and committing it.  Then we can restore from that
sql
> > > > script on startup before calling the stock startup script.
> > > >
> > > > Side benefit is you can see metadata state changes in the git history
and
> > > > manually inspect the sql.
> > > >
> > > >
> > > > On Wed, Mar 27, 2019 at 1:57 PM Pierre Villard <
> > > > pierre.villard.fr@gmail.com>
> > > > wrote:
> > > >
> > > > > Bryan,
> > > > >
> > > > > Thanks for the details. On my side, the main feature I (and others
I
> > > > > suppose) am looking for is the ability to rebuild the metadata DB
from
> > > > the
> > > > > Git repo. But to be honest, this is not a blocker at all: I can live
> > > > with a
> > > > > custom Docker image containing this feature.
> > > > >
> > > > > So, from my point of view, I don't have any issue with #2 but maybe
#1
> > > > > would also give access to experimental features and have interesting
> > > > > feedback from the community. But we would, indeed, need to make it
> > > > crystal
> > > > > clear that the APIs could change in ulterior release(s).
> > > > >
> > > > > Pierre
> > > > >
> > > > >
> > > > >
> > > > > Le mer. 27 mars 2019 à 18:11, Bryan Bende <bbende@gmail.com>
a écrit :
> > > > >
> > > > > > Ryan,
> > > > > >
> > > > > > The short answer to your question is no, there isn't actually
anything
> > > > > > new in the UI. The registry UI is setup in a generic way to
list
> > > > > > "versioned items", so versioned extension bundles will just
> > > > > > automatically show up in the list with versioned flows.
> > > > > >
> > > > > > Let me put together a wiki page that outlines all of the moving
parts
> > > > > > for the extension registry work and where we are currently at.
I'll
> > > > > > send the link on this thread when I have something.
> > > > > >
> > > > > >
> > > > > > For Pierre and others,
> > > > > >
> > > > > > I was thinking more about this discussion of releasing 0.4.0,
and I
> > > > > > think there are two of approaches we could take...
> > > > > >
> > > > > > 1) If there is a desire for the community to release 0.4.0 sooner
> > > > > > rather than later, then we can mark all of the extension bundle
> > > > > > related APIs as experimental, which would then give us the freedom
to
> > > > > > continue evolving them until we have the full extension registry
> > > > > > experience, and likely mark them as stable in 0.5.0, or whichever
> > > > > > future release we decide. The reason for this is that there
will
> > > > > > likely be API changes needed while building out the NiFi side
of
> > > > > > things, and I don't want us to get stuck in a spot where we
can't
> > > > > > change some API since it has been released, and so far the NiFi
side
> > > > > > of the work hasn't been started yet.
> > > > > >
> > > > > > 2) Wait until more of the extension registry work is finalized
and use
> > > > > > that as the driver of when to release 0.4.0, with the obvious
downside
> > > > > > that it may take a bit longer to get a release out which then
holds up
> > > > > > anything else that is not related to extension registry.
> > > > > >
> > > > > > So I guess we just have to decide the driving factor for releasing
> > > > > > 0.4.0... If its the non-extension related features/fixes, then
#1 is
> > > > > > probably the best approach. If its the extension related stuff,
then
> > > > > > I'd vote for #2 since it isn't really ready yet.
> > > > > >
> > > > > > -Bryan
> > > > > >
> > > > > > On Wed, Mar 27, 2019 at 12:01 PM Ryan Hendrickson
> > > > > > <ryan.andrew.hendrickson@gmail.com> wrote:
> > > > > > >
> > > > > > > Bryan,
> > > > > > >    Is there a new GUI part of the registry for the extension
registry
> > > > > > work
> > > > > > > in master?  I compiled 0.4.0-SNAPSHOT and didn't see anything
> > > > > > immediately.
> > > > > > > You mentioned docs and sys admin stuff... Could you point
me in the
> > > > > > > quick-start path if I wanted to try to kick-the-tires a
little on
> > > > this?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Ryan
> > > > > > >
> > > > > > > On Tue, Mar 26, 2019 at 9:57 AM Bryan Bende <bbende@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Mike,
> > > > > > > >
> > > > > > > > All record-oriented components are extensions and
thus can make use
> > > > > of
> > > > > > > > versioned components (i.e. nothing related to records
is baked into
> > > > > > > > the core framework of NiFi).
> > > > > > > >
> > > > > > > > The record API is essentially the module named
> > > > > > > > nifi-record-serialization-services-api which at runtime
is included
> > > > > in
> > > > > > > > nifi-standard-services-api-nar. Any record oriented
processors you
> > > > > > > > build are dependent on a specific version of
> > > > > > > > nifi-standard-services-api-nar since that is where
the Java
> > > > interface
> > > > > > > > will be that your processor was compiled against.
> > > > > > > >
> > > > > > > > Right now, even without extension registry, you could
deploy
> > > > multiple
> > > > > > > > versions of nifi-standard-services-api-nar to a NiFi
instance, and
> > > > > > > > then deploy multiple versions of your record processing
NAR that
> > > > > > > > correspond to the versions of your API NAR.
> > > > > > > >
> > > > > > > > Going forward with extension registry, we probably
want to consider
> > > > > > > > breaking up nifi-standard-services-api-nar into smaller
API NARs,
> > > > as
> > > > > > > > well as nifi-standard-bundle into smaller processor
bundles. For
> > > > > > > > example, there could be a record API NAR and a record
processors
> > > > NAR,
> > > > > > > > that would both be split out of from the standard
NARs.
> > > > > > > >
> > > > > > > > -Bryan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Mar 26, 2019 at 7:55 AM Mike Thomsen <
> > > > mikerthomsen@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Great news about the extension registry! As we
get closer to that
> > > > > > being
> > > > > > > > > ready, I'd like to add in some discussion about
versioning the
> > > > > > record API
> > > > > > > > > separately. A lot of the custom processors we
build now are
> > > > Record
> > > > > > > > > API-based, and it would be great to be able to
decouple them from
> > > > > one
> > > > > > > > > specific release of NiFi.
> > > > > > > > >
> > > > > > > > > On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende <bbende@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Pierre,
> > > > > > > > > >
> > > > > > > > > > I think we are definitely close to an 0.4.0
release. A major
> > > > > chunk
> > > > > > of
> > > > > > > > > > extension registry work has already landed
in master, and I
> > > > still
> > > > > > have
> > > > > > > > one
> > > > > > > > > > other jira that I almost have ready and
wanted to include,
> > > > > > NIFIREG-233
> > > > > > > > for
> > > > > > > > > > generating extensions docs. Plus we probably
need to make a few
> > > > > > > > > > additions/updates to the admin guide.
> > > > > > > > > >
> > > > > > > > > > -Bryan
> > > > > > > > > >
> > > > > > > > > > On Mon, Mar 25, 2019 at 1:33 PM Pierre Villard
<
> > > > > > > > > > pierre.villard.fr@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > All,
> > > > > > > > > > >
> > > > > > > > > > > Some really nice features have been
included since the NiFi
> > > > > > Registry
> > > > > > > > > > 0.3.0
> > > > > > > > > > > release and I'm wondering if it'd be
a good time to consider
> > > > a
> > > > > > 0.4.0
> > > > > > > > > > > release.
> > > > > > > > > > >
> > > > > > > > > > > There is a JIRA tagged for 0.4.0 and
there are 2 opened pull
> > > > > > > > requests,
> > > > > > > > > > plus
> > > > > > > > > > > interesting discussions / JIRAS on
the mailing lists.
> > > > > > > > > > >
> > > > > > > > > > > Are there any reasons to hold off on
a 0.4.0 release?  Are
> > > > > there
> > > > > > > > > > particular
> > > > > > > > > > > JIRAs that the community considers
necessary to have as part
> > > > of
> > > > > > the
> > > > > > > > > > > release?
> > > > > > > > > > >
> > > > > > > > > > > If not, happy to give it a try at performing
RM duties!
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Pierre
> > > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Sent from Gmail Mobile
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >

Mime
View raw message