nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject [DISCUSS] Extension Registry
Date Tue, 13 Nov 2018 17:22:11 GMT
All,

We've needed the elusive extension registry for quite some time now,
and with NiFi Registry I think we are in a good place to make some
progress in this area.

I've started looking into adding "extension bundles" to NiFi Registry
as the next type of versioned item, along side the existing versioned
flows, and I wanted to take a minute to outline how that approach
could work before getting too far into it.

Also, I'd like to focus this discussion on the design and
functionality of the extension registry, and not on how the community
is going to use it. Topics like hosting a central registry, changing
the build process, restructuring the git repo, releasing NARs, etc,
are all important topics, but first we need an extension registry
before we can do any of that :)

Here is a high-level description of what needs to be done...

NiFi Registry

- Add a new type of item called an extension bundle, where each bundle
can contain one ore extensions or APIs

- Support bundles for traditional NiFi (aka NARs) and also bundles for
MiNiFi CPP

- Ability to upload the binary artifact for a bundle and extract the
metadata about the bundle, and metadata about the extensions contained
in the bundle (more on this later)

- Provide a pluggable storage provider for saving the content of each
extension bundle so that we can have different implementations like
local fileysystem, S3, and other object stores

- Provide a REST API for listing and retrieving available bundles,
integrate this into the registry Java client and NiFi CLI

NAR Maven Plugin

- Generate a descriptor for each component in the NAR which will
provide information like the description, tags, restricted or not,
etc.

- These descriptors will be parsed by NiFi Registry when a NAR is
being uploaded so that NiFi Registry will know about the extensions
contained with in the NAR

NiFi

- Provide some type of extension manager experience where users can
search, browse, & install bundles that are available in any of the
registry clients that have been defined

- Introduce a new security policy to control which users are allowed
to access the extension manager

- Installing a bundle should load the NAR and make the extensions
available leveraging the recent work done to auto-load new NARs

- Importing versioned flows from registry should provide an easy way
to install bundles that are required by the flow, but missing from the
local NiFi instance


If anyone has any thoughts or concerns about this approach, please let me know.

Thanks,

Bryan

Mime
View raw message