nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: [DISCUSS] Extension Registry
Date Tue, 13 Nov 2018 18:34:15 GMT
Mark,

I think there are a couple of ways that could be solved, but
ultimately it would be up to how the users choose to setup/manage the
registry, or registries.

The NiFi Registry security model is based around permissions to
buckets (read/write), and all versioned items belong to a bucket. So
you could think of each bucket as a mini extension repository for
which you can control access to specific users and groups, so there
could be a bucket of extensions for each of the NiFi instances in your
example. There can also be multiple registry instances registered with
a given NiFi so extensions can be pulled from multiple registries.

The extensions an instance needs is based on the flows it is running,
the flows already have specific bundle coordinates for every component
in the flow. You can think of it similar to Maven where you declared a
dependency on library foo and the build goes out and gets it for you,
in this case it is a flow that declares a dependency on a bundle.

Mike,

Bundles would need to be uploaded to NiFi Registry (to a specific
bucket) as part of some TBD release process. At a minimum I was
envisioning NiFi CLI commands that can be pointed to a file or a
directory and upload the given bundles to registry. There could be
other options as well, possibly through a Maven plugin to release
directly into registry, or possibly to have a type of extension in
NiFi Registry that actually points to an external location, i.e. all
the NARs that end up in Maven central could somehow be imported into
NiFi Registry, but with pointers back to the content which actually
comes from Maven central. Lot of things to figure out here.

-Bryan
On Tue, Nov 13, 2018 at 1:16 PM Joe Witt <joe.witt@gmail.com> wrote:
>
> Group selection based on tag names for bundles could probably do that.
> Meaning it could be a sorting/filtering mechanism in the NiFi/Registry
> interface perhaps.  Will be good to consider that UX as that
> progresses.
>
> As far as the different environments NiFi instances would certainly be
> able to load only referenced Nars for versioned flows so you'll get
> the optimal set (at runtime) automatically.  Very powerful.
> On Tue, Nov 13, 2018 at 1:12 PM Mark Bean <mark.o.bean@gmail.com> wrote:
> >
> > Joe,
> >
> > I envision the Registry being able to provide a subset of NARs required for
> > a specific NiFi instance. The user may have a relatively small set of NARs
> > required for a NiFi used for basic routing/distribution, and a different
> > more extensive set of NARs required for a more robust NiFi instance which
> > performs various forms of processing/transformations. The grouping I am
> > describing would be a way to select multiple NARs required for a specific
> > NiFi instance.
> >
> > Expanding the scenario a little farther, suppose an integration/test
> > environment defines the group. Then, the production environment can use the
> > group definition to pull (or ensure it possesses) only the relevant NARs
> > necessary.
> >
> > -Mark
> >
> > On Tue, Nov 13, 2018 at 1:00 PM Joe Witt <joe.witt@gmail.com> wrote:
> >
> > > Mark
> > >
> > > Can you describe your use case from the user perspective both for the
> > > entity that would upload the items and demarcate them as a group as
> > > well as the user that would consume those bundles?
> > >
> > > I ask because the point here is that nars are themselves a 'group' in
> > > that they are a logical/contained grouping of extensions.  These can
> > > have relationships to other nars as we know.  And flows are designed
> > > against specific components that come from certain nars for which we
> > > know the precise coordinates.  When importing flows that depend on
> > > these the system will be able to automatically acquire all that it
> > > needs.
> > >
> > > Thanks
> > > On Tue, Nov 13, 2018 at 12:57 PM Mark Bean <mark.o.bean@gmail.com> wrote:
> > > >
> > > > I would like to see a "group" capability in the Registry for NAR bundles.
> > > > Then, users can create their own customized grouping of relevant NARs.
> > > > Managing bundles one at a time will become tedious.
> > > >
> > > > Thanks,
> > > > Mark
> > > >
> > > > On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <joe.witt@gmail.com> wrote:
> > > >
> > > > > Sivaprasanna - yes absolutely.  That is the core point I think of
> > > > > Bryan's first bullet under the 'NiFi' section.
> > > > > On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna <
> > > sivaprasanna246@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > One quick question. With the extension registry, my understanding
is
> > > that
> > > > > > we would try to reduce the overall NiFi size by offloading certain
> > > > > existing
> > > > > > NAR bundles to the extension registry. So are we expecting the
> > > extension
> > > > > > registry to also come up with the ability/mechanism that lets
an
> > > user to
> > > > > > download these bundles ?
> > > > > >
> > > > > > -
> > > > > > Sivaprasanna
> > > > > >
> > > > > > On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <joe.witt@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Bryan
> > > > > > >
> > > > > > > Very exciting to see this under way!!!  We desperately
have to get
> > > our
> > > > > > > core nifi build size way down and make it far easier for
people to
> > > > > > > contribute new processors.  We have a long line of extensions
that
> > > > > > > could be really useful/valuable and this will help unlock
that
> > > while
> > > > > > > improving the user experience tremendously.
> > > > > > >
> > > > > > > For the doc/concerns noted above we should also add/mention
that
> > > the
> > > > > > > relationship between nars (dependencies between nars for
> > > > > > > apis/controller services/parent nars/etc..) we want to
have a way
> > > to
> > > > > > > manage/show that or a user experience for it.  Maybe not
a day-1
> > > thing
> > > > > > > but important to call out.
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Joe
> > > > > > > On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <bbende@gmail.com>
> > > wrote:
> > > > > > > >
> > > > > > > > All,
> > > > > > > >
> > > > > > > > We've needed the elusive extension registry for quite
some time
> > > now,
> > > > > > > > and with NiFi Registry I think we are in a good place
to make
> > > some
> > > > > > > > progress in this area.
> > > > > > > >
> > > > > > > > I've started looking into adding "extension bundles"
to NiFi
> > > Registry
> > > > > > > > as the next type of versioned item, along side the
existing
> > > versioned
> > > > > > > > flows, and I wanted to take a minute to outline how
that approach
> > > > > > > > could work before getting too far into it.
> > > > > > > >
> > > > > > > > Also, I'd like to focus this discussion on the design
and
> > > > > > > > functionality of the extension registry, and not on
how the
> > > community
> > > > > > > > is going to use it. Topics like hosting a central
registry,
> > > changing
> > > > > > > > the build process, restructuring the git repo, releasing
NARs,
> > > etc,
> > > > > > > > are all important topics, but first we need an extension
registry
> > > > > > > > before we can do any of that :)
> > > > > > > >
> > > > > > > > Here is a high-level description of what needs to
be done...
> > > > > > > >
> > > > > > > > NiFi Registry
> > > > > > > >
> > > > > > > > - Add a new type of item called an extension bundle,
where each
> > > > > bundle
> > > > > > > > can contain one ore extensions or APIs
> > > > > > > >
> > > > > > > > - Support bundles for traditional NiFi (aka NARs)
and also
> > > bundles
> > > > > for
> > > > > > > > MiNiFi CPP
> > > > > > > >
> > > > > > > > - Ability to upload the binary artifact for a bundle
and extract
> > > the
> > > > > > > > metadata about the bundle, and metadata about the
extensions
> > > > > contained
> > > > > > > > in the bundle (more on this later)
> > > > > > > >
> > > > > > > > - Provide a pluggable storage provider for saving
the content of
> > > each
> > > > > > > > extension bundle so that we can have different implementations
> > > like
> > > > > > > > local fileysystem, S3, and other object stores
> > > > > > > >
> > > > > > > > - Provide a REST API for listing and retrieving available
> > > bundles,
> > > > > > > > integrate this into the registry Java client and NiFi
CLI
> > > > > > > >
> > > > > > > > NAR Maven Plugin
> > > > > > > >
> > > > > > > > - Generate a descriptor for each component in the
NAR which will
> > > > > > > > provide information like the description, tags, restricted
or
> > > not,
> > > > > > > > etc.
> > > > > > > >
> > > > > > > > - These descriptors will be parsed by NiFi Registry
when a NAR is
> > > > > > > > being uploaded so that NiFi Registry will know about
the
> > > extensions
> > > > > > > > contained with in the NAR
> > > > > > > >
> > > > > > > > NiFi
> > > > > > > >
> > > > > > > > - Provide some type of extension manager experience
where users
> > > can
> > > > > > > > search, browse, & install bundles that are available
in any of
> > > the
> > > > > > > > registry clients that have been defined
> > > > > > > >
> > > > > > > > - Introduce a new security policy to control which
users are
> > > allowed
> > > > > > > > to access the extension manager
> > > > > > > >
> > > > > > > > - Installing a bundle should load the NAR and make
the extensions
> > > > > > > > available leveraging the recent work done to auto-load
new NARs
> > > > > > > >
> > > > > > > > - Importing versioned flows from registry should provide
an easy
> > > way
> > > > > > > > to install bundles that are required by the flow,
but missing
> > > from
> > > > > the
> > > > > > > > local NiFi instance
> > > > > > > >
> > > > > > > >
> > > > > > > > If anyone has any thoughts or concerns about this
approach,
> > > please
> > > > > let
> > > > > > > me know.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Bryan
> > > > > > >
> > > > >
> > >

Mime
View raw message