nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Bean <mark.o.b...@gmail.com>
Subject Re: [DISCUSS] Extension Registry
Date Tue, 13 Nov 2018 18:11:41 GMT
Joe,

I envision the Registry being able to provide a subset of NARs required for
a specific NiFi instance. The user may have a relatively small set of NARs
required for a NiFi used for basic routing/distribution, and a different
more extensive set of NARs required for a more robust NiFi instance which
performs various forms of processing/transformations. The grouping I am
describing would be a way to select multiple NARs required for a specific
NiFi instance.

Expanding the scenario a little farther, suppose an integration/test
environment defines the group. Then, the production environment can use the
group definition to pull (or ensure it possesses) only the relevant NARs
necessary.

-Mark

On Tue, Nov 13, 2018 at 1:00 PM Joe Witt <joe.witt@gmail.com> wrote:

> Mark
>
> Can you describe your use case from the user perspective both for the
> entity that would upload the items and demarcate them as a group as
> well as the user that would consume those bundles?
>
> I ask because the point here is that nars are themselves a 'group' in
> that they are a logical/contained grouping of extensions.  These can
> have relationships to other nars as we know.  And flows are designed
> against specific components that come from certain nars for which we
> know the precise coordinates.  When importing flows that depend on
> these the system will be able to automatically acquire all that it
> needs.
>
> Thanks
> On Tue, Nov 13, 2018 at 12:57 PM Mark Bean <mark.o.bean@gmail.com> wrote:
> >
> > I would like to see a "group" capability in the Registry for NAR bundles.
> > Then, users can create their own customized grouping of relevant NARs.
> > Managing bundles one at a time will become tedious.
> >
> > Thanks,
> > Mark
> >
> > On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <joe.witt@gmail.com> wrote:
> >
> > > Sivaprasanna - yes absolutely.  That is the core point I think of
> > > Bryan's first bullet under the 'NiFi' section.
> > > On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna <
> sivaprasanna246@gmail.com>
> > > wrote:
> > > >
> > > > One quick question. With the extension registry, my understanding is
> that
> > > > we would try to reduce the overall NiFi size by offloading certain
> > > existing
> > > > NAR bundles to the extension registry. So are we expecting the
> extension
> > > > registry to also come up with the ability/mechanism that lets an
> user to
> > > > download these bundles ?
> > > >
> > > > -
> > > > Sivaprasanna
> > > >
> > > > On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <joe.witt@gmail.com>
> wrote:
> > > >
> > > > > Bryan
> > > > >
> > > > > Very exciting to see this under way!!!  We desperately have to get
> our
> > > > > core nifi build size way down and make it far easier for people to
> > > > > contribute new processors.  We have a long line of extensions that
> > > > > could be really useful/valuable and this will help unlock that
> while
> > > > > improving the user experience tremendously.
> > > > >
> > > > > For the doc/concerns noted above we should also add/mention that
> the
> > > > > relationship between nars (dependencies between nars for
> > > > > apis/controller services/parent nars/etc..) we want to have a way
> to
> > > > > manage/show that or a user experience for it.  Maybe not a day-1
> thing
> > > > > but important to call out.
> > > > >
> > > > > Thanks!
> > > > > Joe
> > > > > On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <bbende@gmail.com>
> wrote:
> > > > > >
> > > > > > All,
> > > > > >
> > > > > > We've needed the elusive extension registry for quite some time
> now,
> > > > > > and with NiFi Registry I think we are in a good place to make
> some
> > > > > > progress in this area.
> > > > > >
> > > > > > I've started looking into adding "extension bundles" to NiFi
> Registry
> > > > > > as the next type of versioned item, along side the existing
> versioned
> > > > > > flows, and I wanted to take a minute to outline how that approach
> > > > > > could work before getting too far into it.
> > > > > >
> > > > > > Also, I'd like to focus this discussion on the design and
> > > > > > functionality of the extension registry, and not on how the
> community
> > > > > > is going to use it. Topics like hosting a central registry,
> changing
> > > > > > the build process, restructuring the git repo, releasing NARs,
> etc,
> > > > > > are all important topics, but first we need an extension registry
> > > > > > before we can do any of that :)
> > > > > >
> > > > > > Here is a high-level description of what needs to be done...
> > > > > >
> > > > > > NiFi Registry
> > > > > >
> > > > > > - Add a new type of item called an extension bundle, where each
> > > bundle
> > > > > > can contain one ore extensions or APIs
> > > > > >
> > > > > > - Support bundles for traditional NiFi (aka NARs) and also
> bundles
> > > for
> > > > > > MiNiFi CPP
> > > > > >
> > > > > > - Ability to upload the binary artifact for a bundle and extract
> the
> > > > > > metadata about the bundle, and metadata about the extensions
> > > contained
> > > > > > in the bundle (more on this later)
> > > > > >
> > > > > > - Provide a pluggable storage provider for saving the content
of
> each
> > > > > > extension bundle so that we can have different implementations
> like
> > > > > > local fileysystem, S3, and other object stores
> > > > > >
> > > > > > - Provide a REST API for listing and retrieving available
> bundles,
> > > > > > integrate this into the registry Java client and NiFi CLI
> > > > > >
> > > > > > NAR Maven Plugin
> > > > > >
> > > > > > - Generate a descriptor for each component in the NAR which
will
> > > > > > provide information like the description, tags, restricted or
> not,
> > > > > > etc.
> > > > > >
> > > > > > - These descriptors will be parsed by NiFi Registry when a NAR
is
> > > > > > being uploaded so that NiFi Registry will know about the
> extensions
> > > > > > contained with in the NAR
> > > > > >
> > > > > > NiFi
> > > > > >
> > > > > > - Provide some type of extension manager experience where users
> can
> > > > > > search, browse, & install bundles that are available in
any of
> the
> > > > > > registry clients that have been defined
> > > > > >
> > > > > > - Introduce a new security policy to control which users are
> allowed
> > > > > > to access the extension manager
> > > > > >
> > > > > > - Installing a bundle should load the NAR and make the extensions
> > > > > > available leveraging the recent work done to auto-load new NARs
> > > > > >
> > > > > > - Importing versioned flows from registry should provide an
easy
> way
> > > > > > to install bundles that are required by the flow, but missing
> from
> > > the
> > > > > > local NiFi instance
> > > > > >
> > > > > >
> > > > > > If anyone has any thoughts or concerns about this approach,
> please
> > > let
> > > > > me know.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Bryan
> > > > >
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message