nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Moser <moser...@gmail.com>
Subject Re: [DISCUSS] Extension Registry
Date Tue, 13 Nov 2018 19:32:14 GMT
I have thought about this in the past, too.  Here's a scenario where I
could never really lay down an approach I was happy with.

Consider that a NiFi user searches the NiFi registry and finds the PutMongo
processor.  Registry knows the PutMongo processor is in the
nifi-mongodb-nar, and through its Nar-Dependency-Id that it has a
dependency on a controller service interface in
nifi-mongodb-client-service-api-nar.  Great, the user can then download and
install those two nars.  How would we then suggest that the user also needs
a MongoDBClientService controller service implementation, such as that in
the nifi-mongodb-services-nar?

I'm not looking for an answer now, of course, but I just wanted to feed the
discussion.

Thanks,
-- Mike


On Tue, Nov 13, 2018 at 1:34 PM Bryan Bende <bbende@gmail.com> wrote:

> Mark,
>
> I think there are a couple of ways that could be solved, but
> ultimately it would be up to how the users choose to setup/manage the
> registry, or registries.
>
> The NiFi Registry security model is based around permissions to
> buckets (read/write), and all versioned items belong to a bucket. So
> you could think of each bucket as a mini extension repository for
> which you can control access to specific users and groups, so there
> could be a bucket of extensions for each of the NiFi instances in your
> example. There can also be multiple registry instances registered with
> a given NiFi so extensions can be pulled from multiple registries.
>
> The extensions an instance needs is based on the flows it is running,
> the flows already have specific bundle coordinates for every component
> in the flow. You can think of it similar to Maven where you declared a
> dependency on library foo and the build goes out and gets it for you,
> in this case it is a flow that declares a dependency on a bundle.
>
> Mike,
>
> Bundles would need to be uploaded to NiFi Registry (to a specific
> bucket) as part of some TBD release process. At a minimum I was
> envisioning NiFi CLI commands that can be pointed to a file or a
> directory and upload the given bundles to registry. There could be
> other options as well, possibly through a Maven plugin to release
> directly into registry, or possibly to have a type of extension in
> NiFi Registry that actually points to an external location, i.e. all
> the NARs that end up in Maven central could somehow be imported into
> NiFi Registry, but with pointers back to the content which actually
> comes from Maven central. Lot of things to figure out here.
>
> -Bryan
> On Tue, Nov 13, 2018 at 1:16 PM Joe Witt <joe.witt@gmail.com> wrote:
> >
> > Group selection based on tag names for bundles could probably do that.
> > Meaning it could be a sorting/filtering mechanism in the NiFi/Registry
> > interface perhaps.  Will be good to consider that UX as that
> > progresses.
> >
> > As far as the different environments NiFi instances would certainly be
> > able to load only referenced Nars for versioned flows so you'll get
> > the optimal set (at runtime) automatically.  Very powerful.
> > On Tue, Nov 13, 2018 at 1:12 PM Mark Bean <mark.o.bean@gmail.com> wrote:
> > >
> > > Joe,
> > >
> > > I envision the Registry being able to provide a subset of NARs
> required for
> > > a specific NiFi instance. The user may have a relatively small set of
> NARs
> > > required for a NiFi used for basic routing/distribution, and a
> different
> > > more extensive set of NARs required for a more robust NiFi instance
> which
> > > performs various forms of processing/transformations. The grouping I am
> > > describing would be a way to select multiple NARs required for a
> specific
> > > NiFi instance.
> > >
> > > Expanding the scenario a little farther, suppose an integration/test
> > > environment defines the group. Then, the production environment can
> use the
> > > group definition to pull (or ensure it possesses) only the relevant
> NARs
> > > necessary.
> > >
> > > -Mark
> > >
> > > On Tue, Nov 13, 2018 at 1:00 PM Joe Witt <joe.witt@gmail.com> wrote:
> > >
> > > > Mark
> > > >
> > > > Can you describe your use case from the user perspective both for the
> > > > entity that would upload the items and demarcate them as a group as
> > > > well as the user that would consume those bundles?
> > > >
> > > > I ask because the point here is that nars are themselves a 'group' in
> > > > that they are a logical/contained grouping of extensions.  These can
> > > > have relationships to other nars as we know.  And flows are designed
> > > > against specific components that come from certain nars for which we
> > > > know the precise coordinates.  When importing flows that depend on
> > > > these the system will be able to automatically acquire all that it
> > > > needs.
> > > >
> > > > Thanks
> > > > On Tue, Nov 13, 2018 at 12:57 PM Mark Bean <mark.o.bean@gmail.com>
> wrote:
> > > > >
> > > > > I would like to see a "group" capability in the Registry for NAR
> bundles.
> > > > > Then, users can create their own customized grouping of relevant
> NARs.
> > > > > Managing bundles one at a time will become tedious.
> > > > >
> > > > > Thanks,
> > > > > Mark
> > > > >
> > > > > On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <joe.witt@gmail.com>
> wrote:
> > > > >
> > > > > > Sivaprasanna - yes absolutely.  That is the core point I think
of
> > > > > > Bryan's first bullet under the 'NiFi' section.
> > > > > > On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna <
> > > > sivaprasanna246@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > One quick question. With the extension registry, my
> understanding is
> > > > that
> > > > > > > we would try to reduce the overall NiFi size by offloading
> certain
> > > > > > existing
> > > > > > > NAR bundles to the extension registry. So are we expecting
the
> > > > extension
> > > > > > > registry to also come up with the ability/mechanism that
lets
> an
> > > > user to
> > > > > > > download these bundles ?
> > > > > > >
> > > > > > > -
> > > > > > > Sivaprasanna
> > > > > > >
> > > > > > > On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <joe.witt@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Bryan
> > > > > > > >
> > > > > > > > Very exciting to see this under way!!!  We desperately
have
> to get
> > > > our
> > > > > > > > core nifi build size way down and make it far easier
for
> people to
> > > > > > > > contribute new processors.  We have a long line of
> extensions that
> > > > > > > > could be really useful/valuable and this will help
unlock
> that
> > > > while
> > > > > > > > improving the user experience tremendously.
> > > > > > > >
> > > > > > > > For the doc/concerns noted above we should also add/mention
> that
> > > > the
> > > > > > > > relationship between nars (dependencies between nars
for
> > > > > > > > apis/controller services/parent nars/etc..) we want
to have
> a way
> > > > to
> > > > > > > > manage/show that or a user experience for it.  Maybe
not a
> day-1
> > > > thing
> > > > > > > > but important to call out.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > Joe
> > > > > > > > On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <
> bbende@gmail.com>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > All,
> > > > > > > > >
> > > > > > > > > We've needed the elusive extension registry for
quite some
> time
> > > > now,
> > > > > > > > > and with NiFi Registry I think we are in a good
place to
> make
> > > > some
> > > > > > > > > progress in this area.
> > > > > > > > >
> > > > > > > > > I've started looking into adding "extension bundles"
to
> NiFi
> > > > Registry
> > > > > > > > > as the next type of versioned item, along side
the existing
> > > > versioned
> > > > > > > > > flows, and I wanted to take a minute to outline
how that
> approach
> > > > > > > > > could work before getting too far into it.
> > > > > > > > >
> > > > > > > > > Also, I'd like to focus this discussion on the
design and
> > > > > > > > > functionality of the extension registry, and
not on how the
> > > > community
> > > > > > > > > is going to use it. Topics like hosting a central
registry,
> > > > changing
> > > > > > > > > the build process, restructuring the git repo,
releasing
> NARs,
> > > > etc,
> > > > > > > > > are all important topics, but first we need an
extension
> registry
> > > > > > > > > before we can do any of that :)
> > > > > > > > >
> > > > > > > > > Here is a high-level description of what needs
to be
> done...
> > > > > > > > >
> > > > > > > > > NiFi Registry
> > > > > > > > >
> > > > > > > > > - Add a new type of item called an extension
bundle, where
> each
> > > > > > bundle
> > > > > > > > > can contain one ore extensions or APIs
> > > > > > > > >
> > > > > > > > > - Support bundles for traditional NiFi (aka NARs)
and also
> > > > bundles
> > > > > > for
> > > > > > > > > MiNiFi CPP
> > > > > > > > >
> > > > > > > > > - Ability to upload the binary artifact for a
bundle and
> extract
> > > > the
> > > > > > > > > metadata about the bundle, and metadata about
the
> extensions
> > > > > > contained
> > > > > > > > > in the bundle (more on this later)
> > > > > > > > >
> > > > > > > > > - Provide a pluggable storage provider for saving
the
> content of
> > > > each
> > > > > > > > > extension bundle so that we can have different
> implementations
> > > > like
> > > > > > > > > local fileysystem, S3, and other object stores
> > > > > > > > >
> > > > > > > > > - Provide a REST API for listing and retrieving
available
> > > > bundles,
> > > > > > > > > integrate this into the registry Java client
and NiFi CLI
> > > > > > > > >
> > > > > > > > > NAR Maven Plugin
> > > > > > > > >
> > > > > > > > > - Generate a descriptor for each component in
the NAR
> which will
> > > > > > > > > provide information like the description, tags,
restricted
> or
> > > > not,
> > > > > > > > > etc.
> > > > > > > > >
> > > > > > > > > - These descriptors will be parsed by NiFi Registry
when a
> NAR is
> > > > > > > > > being uploaded so that NiFi Registry will know
about the
> > > > extensions
> > > > > > > > > contained with in the NAR
> > > > > > > > >
> > > > > > > > > NiFi
> > > > > > > > >
> > > > > > > > > - Provide some type of extension manager experience
where
> users
> > > > can
> > > > > > > > > search, browse, & install bundles that are
available in
> any of
> > > > the
> > > > > > > > > registry clients that have been defined
> > > > > > > > >
> > > > > > > > > - Introduce a new security policy to control
which users
> are
> > > > allowed
> > > > > > > > > to access the extension manager
> > > > > > > > >
> > > > > > > > > - Installing a bundle should load the NAR and
make the
> extensions
> > > > > > > > > available leveraging the recent work done to
auto-load new
> NARs
> > > > > > > > >
> > > > > > > > > - Importing versioned flows from registry should
provide
> an easy
> > > > way
> > > > > > > > > to install bundles that are required by the flow,
but
> missing
> > > > from
> > > > > > the
> > > > > > > > > local NiFi instance
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > If anyone has any thoughts or concerns about
this approach,
> > > > please
> > > > > > let
> > > > > > > > me know.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Bryan
> > > > > > > >
> > > > > >
> > > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message