nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Thomsen <mikerthom...@gmail.com>
Subject Re: [DISCUSS] Extension Registry
Date Tue, 13 Nov 2018 18:08:07 GMT
What are the plans for enabling the registry/NiFi to pull NARs and deploy
them? Off the top of my head, piggybacking off of the Maven libraries/repos
might be a good way to go there if there are no other plans.

Private repositories should be baked in from the start here. Ideal would be
something where DevOps could have Jenkins set up a job that builds a
bundle, "deploys it" and the extension registry is informed that a new
NAR/set of NARs are there. If we go with hot-swappable ones, might also be
a good idea to borrow from Maven there by allowing different snapshot
builds to coexist so users can tell the Registry to swap one build for
another.

Mike

On Tue, Nov 13, 2018 at 1:00 PM Joe Witt <joe.witt@gmail.com> wrote:

> Mark
>
> Can you describe your use case from the user perspective both for the
> entity that would upload the items and demarcate them as a group as
> well as the user that would consume those bundles?
>
> I ask because the point here is that nars are themselves a 'group' in
> that they are a logical/contained grouping of extensions.  These can
> have relationships to other nars as we know.  And flows are designed
> against specific components that come from certain nars for which we
> know the precise coordinates.  When importing flows that depend on
> these the system will be able to automatically acquire all that it
> needs.
>
> Thanks
> On Tue, Nov 13, 2018 at 12:57 PM Mark Bean <mark.o.bean@gmail.com> wrote:
> >
> > I would like to see a "group" capability in the Registry for NAR bundles.
> > Then, users can create their own customized grouping of relevant NARs.
> > Managing bundles one at a time will become tedious.
> >
> > Thanks,
> > Mark
> >
> > On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <joe.witt@gmail.com> wrote:
> >
> > > Sivaprasanna - yes absolutely.  That is the core point I think of
> > > Bryan's first bullet under the 'NiFi' section.
> > > On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna <
> sivaprasanna246@gmail.com>
> > > wrote:
> > > >
> > > > One quick question. With the extension registry, my understanding is
> that
> > > > we would try to reduce the overall NiFi size by offloading certain
> > > existing
> > > > NAR bundles to the extension registry. So are we expecting the
> extension
> > > > registry to also come up with the ability/mechanism that lets an
> user to
> > > > download these bundles ?
> > > >
> > > > -
> > > > Sivaprasanna
> > > >
> > > > On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <joe.witt@gmail.com>
> wrote:
> > > >
> > > > > Bryan
> > > > >
> > > > > Very exciting to see this under way!!!  We desperately have to get
> our
> > > > > core nifi build size way down and make it far easier for people to
> > > > > contribute new processors.  We have a long line of extensions that
> > > > > could be really useful/valuable and this will help unlock that
> while
> > > > > improving the user experience tremendously.
> > > > >
> > > > > For the doc/concerns noted above we should also add/mention that
> the
> > > > > relationship between nars (dependencies between nars for
> > > > > apis/controller services/parent nars/etc..) we want to have a way
> to
> > > > > manage/show that or a user experience for it.  Maybe not a day-1
> thing
> > > > > but important to call out.
> > > > >
> > > > > Thanks!
> > > > > Joe
> > > > > On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <bbende@gmail.com>
> wrote:
> > > > > >
> > > > > > All,
> > > > > >
> > > > > > We've needed the elusive extension registry for quite some time
> now,
> > > > > > and with NiFi Registry I think we are in a good place to make
> some
> > > > > > progress in this area.
> > > > > >
> > > > > > I've started looking into adding "extension bundles" to NiFi
> Registry
> > > > > > as the next type of versioned item, along side the existing
> versioned
> > > > > > flows, and I wanted to take a minute to outline how that approach
> > > > > > could work before getting too far into it.
> > > > > >
> > > > > > Also, I'd like to focus this discussion on the design and
> > > > > > functionality of the extension registry, and not on how the
> community
> > > > > > is going to use it. Topics like hosting a central registry,
> changing
> > > > > > the build process, restructuring the git repo, releasing NARs,
> etc,
> > > > > > are all important topics, but first we need an extension registry
> > > > > > before we can do any of that :)
> > > > > >
> > > > > > Here is a high-level description of what needs to be done...
> > > > > >
> > > > > > NiFi Registry
> > > > > >
> > > > > > - Add a new type of item called an extension bundle, where each
> > > bundle
> > > > > > can contain one ore extensions or APIs
> > > > > >
> > > > > > - Support bundles for traditional NiFi (aka NARs) and also
> bundles
> > > for
> > > > > > MiNiFi CPP
> > > > > >
> > > > > > - Ability to upload the binary artifact for a bundle and extract
> the
> > > > > > metadata about the bundle, and metadata about the extensions
> > > contained
> > > > > > in the bundle (more on this later)
> > > > > >
> > > > > > - Provide a pluggable storage provider for saving the content
of
> each
> > > > > > extension bundle so that we can have different implementations
> like
> > > > > > local fileysystem, S3, and other object stores
> > > > > >
> > > > > > - Provide a REST API for listing and retrieving available
> bundles,
> > > > > > integrate this into the registry Java client and NiFi CLI
> > > > > >
> > > > > > NAR Maven Plugin
> > > > > >
> > > > > > - Generate a descriptor for each component in the NAR which
will
> > > > > > provide information like the description, tags, restricted or
> not,
> > > > > > etc.
> > > > > >
> > > > > > - These descriptors will be parsed by NiFi Registry when a NAR
is
> > > > > > being uploaded so that NiFi Registry will know about the
> extensions
> > > > > > contained with in the NAR
> > > > > >
> > > > > > NiFi
> > > > > >
> > > > > > - Provide some type of extension manager experience where users
> can
> > > > > > search, browse, & install bundles that are available in
any of
> the
> > > > > > registry clients that have been defined
> > > > > >
> > > > > > - Introduce a new security policy to control which users are
> allowed
> > > > > > to access the extension manager
> > > > > >
> > > > > > - Installing a bundle should load the NAR and make the extensions
> > > > > > available leveraging the recent work done to auto-load new NARs
> > > > > >
> > > > > > - Importing versioned flows from registry should provide an
easy
> way
> > > > > > to install bundles that are required by the flow, but missing
> from
> > > the
> > > > > > local NiFi instance
> > > > > >
> > > > > >
> > > > > > If anyone has any thoughts or concerns about this approach,
> please
> > > let
> > > > > me know.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Bryan
> > > > >
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message