nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy LoPresto <alopresto.apa...@gmail.com>
Subject Re: [DISCUSS] Extension Registry
Date Tue, 13 Nov 2018 20:06:22 GMT
From a security perspective, I’d like to see attention paid to the following topics:

* verifiable checksums over extensions/bundles to ensure the code being deployed from a remote
system is trusted
* fuzzy hashing (complementary) to provide “change factor” on new version of the same
extension/bundle
* cryptographic signatures from a trusted entity on extensions/bundles to allow for verification
even over untrusted transmission
* mechanisms to verify the available signatures automatically within NiFi rather than requiring
external/manual verification 
* the same mutual authentication via TLS we provide for other services
* the ability to configure multiple registries to allow for redundancy, load balancing, mitigation
of denial of service, etc. 

Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Nov 14, 2018, at 06:55, Andrew Grande <aperepel@gmail.com> wrote:
> 
> I would like to see a clear separation of blob and metadata storage. Most
> often you'd see some object storage being already distributed and
> replicated, let's think about an easy way to backup or migrate the metadata
> between registry instances.
> 
> Andrew
> 
>> On Tue, Nov 13, 2018, 11:32 AM Michael Moser <moser.mw@gmail.com> wrote:
>> 
>> I have thought about this in the past, too.  Here's a scenario where I
>> could never really lay down an approach I was happy with.
>> 
>> Consider that a NiFi user searches the NiFi registry and finds the PutMongo
>> processor.  Registry knows the PutMongo processor is in the
>> nifi-mongodb-nar, and through its Nar-Dependency-Id that it has a
>> dependency on a controller service interface in
>> nifi-mongodb-client-service-api-nar.  Great, the user can then download and
>> install those two nars.  How would we then suggest that the user also needs
>> a MongoDBClientService controller service implementation, such as that in
>> the nifi-mongodb-services-nar?
>> 
>> I'm not looking for an answer now, of course, but I just wanted to feed the
>> discussion.
>> 
>> Thanks,
>> -- Mike
>> 
>> 
>>> On Tue, Nov 13, 2018 at 1:34 PM Bryan Bende <bbende@gmail.com> wrote:
>>> 
>>> Mark,
>>> 
>>> I think there are a couple of ways that could be solved, but
>>> ultimately it would be up to how the users choose to setup/manage the
>>> registry, or registries.
>>> 
>>> The NiFi Registry security model is based around permissions to
>>> buckets (read/write), and all versioned items belong to a bucket. So
>>> you could think of each bucket as a mini extension repository for
>>> which you can control access to specific users and groups, so there
>>> could be a bucket of extensions for each of the NiFi instances in your
>>> example. There can also be multiple registry instances registered with
>>> a given NiFi so extensions can be pulled from multiple registries.
>>> 
>>> The extensions an instance needs is based on the flows it is running,
>>> the flows already have specific bundle coordinates for every component
>>> in the flow. You can think of it similar to Maven where you declared a
>>> dependency on library foo and the build goes out and gets it for you,
>>> in this case it is a flow that declares a dependency on a bundle.
>>> 
>>> Mike,
>>> 
>>> Bundles would need to be uploaded to NiFi Registry (to a specific
>>> bucket) as part of some TBD release process. At a minimum I was
>>> envisioning NiFi CLI commands that can be pointed to a file or a
>>> directory and upload the given bundles to registry. There could be
>>> other options as well, possibly through a Maven plugin to release
>>> directly into registry, or possibly to have a type of extension in
>>> NiFi Registry that actually points to an external location, i.e. all
>>> the NARs that end up in Maven central could somehow be imported into
>>> NiFi Registry, but with pointers back to the content which actually
>>> comes from Maven central. Lot of things to figure out here.
>>> 
>>> -Bryan
>>>> On Tue, Nov 13, 2018 at 1:16 PM Joe Witt <joe.witt@gmail.com> wrote:
>>>> 
>>>> Group selection based on tag names for bundles could probably do that.
>>>> Meaning it could be a sorting/filtering mechanism in the NiFi/Registry
>>>> interface perhaps.  Will be good to consider that UX as that
>>>> progresses.
>>>> 
>>>> As far as the different environments NiFi instances would certainly be
>>>> able to load only referenced Nars for versioned flows so you'll get
>>>> the optimal set (at runtime) automatically.  Very powerful.
>>>> On Tue, Nov 13, 2018 at 1:12 PM Mark Bean <mark.o.bean@gmail.com>
>> wrote:
>>>>> 
>>>>> Joe,
>>>>> 
>>>>> I envision the Registry being able to provide a subset of NARs
>>> required for
>>>>> a specific NiFi instance. The user may have a relatively small set of
>>> NARs
>>>>> required for a NiFi used for basic routing/distribution, and a
>>> different
>>>>> more extensive set of NARs required for a more robust NiFi instance
>>> which
>>>>> performs various forms of processing/transformations. The grouping I
>> am
>>>>> describing would be a way to select multiple NARs required for a
>>> specific
>>>>> NiFi instance.
>>>>> 
>>>>> Expanding the scenario a little farther, suppose an integration/test
>>>>> environment defines the group. Then, the production environment can
>>> use the
>>>>> group definition to pull (or ensure it possesses) only the relevant
>>> NARs
>>>>> necessary.
>>>>> 
>>>>> -Mark
>>>>> 
>>>>>> On Tue, Nov 13, 2018 at 1:00 PM Joe Witt <joe.witt@gmail.com>
wrote:
>>>>>> 
>>>>>> Mark
>>>>>> 
>>>>>> Can you describe your use case from the user perspective both for
>> the
>>>>>> entity that would upload the items and demarcate them as a group
as
>>>>>> well as the user that would consume those bundles?
>>>>>> 
>>>>>> I ask because the point here is that nars are themselves a 'group'
>> in
>>>>>> that they are a logical/contained grouping of extensions.  These
>> can
>>>>>> have relationships to other nars as we know.  And flows are
>> designed
>>>>>> against specific components that come from certain nars for which
>> we
>>>>>> know the precise coordinates.  When importing flows that depend on
>>>>>> these the system will be able to automatically acquire all that it
>>>>>> needs.
>>>>>> 
>>>>>> Thanks
>>>>>> On Tue, Nov 13, 2018 at 12:57 PM Mark Bean <mark.o.bean@gmail.com>
>>> wrote:
>>>>>>> 
>>>>>>> I would like to see a "group" capability in the Registry for
NAR
>>> bundles.
>>>>>>> Then, users can create their own customized grouping of relevant
>>> NARs.
>>>>>>> Managing bundles one at a time will become tedious.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Mark
>>>>>>> 
>>>>>>> On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <joe.witt@gmail.com>
>>> wrote:
>>>>>>> 
>>>>>>>> Sivaprasanna - yes absolutely.  That is the core point I
think
>> of
>>>>>>>> Bryan's first bullet under the 'NiFi' section.
>>>>>>>> On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna <
>>>>>> sivaprasanna246@gmail.com>
>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> One quick question. With the extension registry, my
>>> understanding is
>>>>>> that
>>>>>>>>> we would try to reduce the overall NiFi size by offloading
>>> certain
>>>>>>>> existing
>>>>>>>>> NAR bundles to the extension registry. So are we expecting
>> the
>>>>>> extension
>>>>>>>>> registry to also come up with the ability/mechanism that
lets
>>> an
>>>>>> user to
>>>>>>>>> download these bundles ?
>>>>>>>>> 
>>>>>>>>> -
>>>>>>>>> Sivaprasanna
>>>>>>>>> 
>>>>>>>>> On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <
>> joe.witt@gmail.com>
>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Bryan
>>>>>>>>>> 
>>>>>>>>>> Very exciting to see this under way!!!  We desperately
have
>>> to get
>>>>>> our
>>>>>>>>>> core nifi build size way down and make it far easier
for
>>> people to
>>>>>>>>>> contribute new processors.  We have a long line of
>>> extensions that
>>>>>>>>>> could be really useful/valuable and this will help
unlock
>>> that
>>>>>> while
>>>>>>>>>> improving the user experience tremendously.
>>>>>>>>>> 
>>>>>>>>>> For the doc/concerns noted above we should also add/mention
>>> that
>>>>>> the
>>>>>>>>>> relationship between nars (dependencies between nars
for
>>>>>>>>>> apis/controller services/parent nars/etc..) we want
to have
>>> a way
>>>>>> to
>>>>>>>>>> manage/show that or a user experience for it.  Maybe
not a
>>> day-1
>>>>>> thing
>>>>>>>>>> but important to call out.
>>>>>>>>>> 
>>>>>>>>>> Thanks!
>>>>>>>>>> Joe
>>>>>>>>>> On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <
>>> bbende@gmail.com>
>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> All,
>>>>>>>>>>> 
>>>>>>>>>>> We've needed the elusive extension registry for
quite
>> some
>>> time
>>>>>> now,
>>>>>>>>>>> and with NiFi Registry I think we are in a good
place to
>>> make
>>>>>> some
>>>>>>>>>>> progress in this area.
>>>>>>>>>>> 
>>>>>>>>>>> I've started looking into adding "extension bundles"
to
>>> NiFi
>>>>>> Registry
>>>>>>>>>>> as the next type of versioned item, along side
the
>> existing
>>>>>> versioned
>>>>>>>>>>> flows, and I wanted to take a minute to outline
how that
>>> approach
>>>>>>>>>>> could work before getting too far into it.
>>>>>>>>>>> 
>>>>>>>>>>> Also, I'd like to focus this discussion on the
design and
>>>>>>>>>>> functionality of the extension registry, and
not on how
>> the
>>>>>> community
>>>>>>>>>>> is going to use it. Topics like hosting a central
>> registry,
>>>>>> changing
>>>>>>>>>>> the build process, restructuring the git repo,
releasing
>>> NARs,
>>>>>> etc,
>>>>>>>>>>> are all important topics, but first we need an
extension
>>> registry
>>>>>>>>>>> before we can do any of that :)
>>>>>>>>>>> 
>>>>>>>>>>> Here is a high-level description of what needs
to be
>>> done...
>>>>>>>>>>> 
>>>>>>>>>>> NiFi Registry
>>>>>>>>>>> 
>>>>>>>>>>> - Add a new type of item called an extension
bundle,
>> where
>>> each
>>>>>>>> bundle
>>>>>>>>>>> can contain one ore extensions or APIs
>>>>>>>>>>> 
>>>>>>>>>>> - Support bundles for traditional NiFi (aka NARs)
and
>> also
>>>>>> bundles
>>>>>>>> for
>>>>>>>>>>> MiNiFi CPP
>>>>>>>>>>> 
>>>>>>>>>>> - Ability to upload the binary artifact for a
bundle and
>>> extract
>>>>>> the
>>>>>>>>>>> metadata about the bundle, and metadata about
the
>>> extensions
>>>>>>>> contained
>>>>>>>>>>> in the bundle (more on this later)
>>>>>>>>>>> 
>>>>>>>>>>> - Provide a pluggable storage provider for saving
the
>>> content of
>>>>>> each
>>>>>>>>>>> extension bundle so that we can have different
>>> implementations
>>>>>> like
>>>>>>>>>>> local fileysystem, S3, and other object stores
>>>>>>>>>>> 
>>>>>>>>>>> - Provide a REST API for listing and retrieving
available
>>>>>> bundles,
>>>>>>>>>>> integrate this into the registry Java client
and NiFi CLI
>>>>>>>>>>> 
>>>>>>>>>>> NAR Maven Plugin
>>>>>>>>>>> 
>>>>>>>>>>> - Generate a descriptor for each component in
the NAR
>>> which will
>>>>>>>>>>> provide information like the description, tags,
>> restricted
>>> or
>>>>>> not,
>>>>>>>>>>> etc.
>>>>>>>>>>> 
>>>>>>>>>>> - These descriptors will be parsed by NiFi Registry
when
>> a
>>> NAR is
>>>>>>>>>>> being uploaded so that NiFi Registry will know
about the
>>>>>> extensions
>>>>>>>>>>> contained with in the NAR
>>>>>>>>>>> 
>>>>>>>>>>> NiFi
>>>>>>>>>>> 
>>>>>>>>>>> - Provide some type of extension manager experience
where
>>> users
>>>>>> can
>>>>>>>>>>> search, browse, & install bundles that are
available in
>>> any of
>>>>>> the
>>>>>>>>>>> registry clients that have been defined
>>>>>>>>>>> 
>>>>>>>>>>> - Introduce a new security policy to control
which users
>>> are
>>>>>> allowed
>>>>>>>>>>> to access the extension manager
>>>>>>>>>>> 
>>>>>>>>>>> - Installing a bundle should load the NAR and
make the
>>> extensions
>>>>>>>>>>> available leveraging the recent work done to
auto-load
>> new
>>> NARs
>>>>>>>>>>> 
>>>>>>>>>>> - Importing versioned flows from registry should
provide
>>> an easy
>>>>>> way
>>>>>>>>>>> to install bundles that are required by the flow,
but
>>> missing
>>>>>> from
>>>>>>>> the
>>>>>>>>>>> local NiFi instance
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> If anyone has any thoughts or concerns about
this
>> approach,
>>>>>> please
>>>>>>>> let
>>>>>>>>>> me know.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> 
>>>>>>>>>>> Bryan
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>> 
>> 

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message