nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: NiFi Registry feature branch workflow (possibly via Git-Backed MetadataService, FlowPersistenceProvider)
Date Tue, 05 Mar 2019 16:58:03 GMT
Bryan,

The idea of branching to test changes is definitely interesting. A
couple of thoughts I had while thinking about it...

1) The FlowPersistenceProvider is an extension point, so how would
branching work if someone is not using the GitFlowPersistenceProvider?
I guess it could be disable if FlowPersistenceProvider is not an
instance of GitFlowPersistenceProvider (or some Branchable interface)?

Personally I prefer to treat the git persistence provider as just
another binary storage for flows, with the added bonus that if you use
tools like GitHub you get some nice visuals and an automatic backup.
For most functionality I would prefer it still happen through our
applications. For example, many users want to diff flows using GitHub,
but it would be preferable to build a proper diff capability into NiFi
and NiFi Registry that worked regardless of the persistence provider
(there is already a REST end-point, we just need some UI work).

2) As far as the metadata service... One idea we thought of early on
was to have a metadata service for each type of versioned item, so for
example flows could have there metadata one place, and then extensions
somewhere else, but then the issue is that buckets can have multiple
types of items so there still has to be something that ties together
the knowledge of all items across all buckets, which is currently what
the metadata service is.

I'm also not exactly sure how we'd implement all of the current access
patterns in the MetadataService [1] backed by git. There are a
significant number of methods to implement that have been built around
the idea of SQL access since there was no intention of this being
pluggable, plus it is continuing to grow with all of the extension
registry work, there are about 7 new DB tables so far to support
extension registry.

3) What would the UX be in NiFi UI and NiFi Registry UI?

>From the description of the idea, I'm not totally sure if the
branching concept would be completely behind the scenes or not, but if
it was going to be a first class concept then we'd need to put some
design and thought into how it fits into the UX.

Thanks,

Bryan

[1] https://github.com/apache/nifi-registry/blob/master/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/service/MetadataService.java

On Tue, Mar 5, 2019 at 11:08 AM Bryan Rosander <bryanrosander@gmail.com> wrote:
>
> Hi all,
>
> We're trying to implement a feature branch workflow with NiFi Registry.
> The basic idea is that you'd be able to have one or more branches off of
> master and then the capability to merge changes back in when they're ready.
>
> Our flow has many versioned process groups that are nested several layers
> deep.
>
> There are a couple of approaches that we're thinking about:
>
> NiPyApi [1] (hacky but should work without registry changes):
> 1. Keep track of versions when branch was created
> 2. Use NiPyApi to compile a list of changed flows since the branch
> 3. Apply changes to master registry from the bottom process group(s) up,
> updating version numbers of child pgs as we go, writing out to some sort of
> patch file
> 4. Verify patch file looks right, use NiPyApi to apply it (could do peer
> review as part of this)
>
> Git-Based:
> Create a git-backed MetadataService that can coexist with
> the GitFlowPersistenceProvider (share a repository, not mess with each
> others' metadata)
>
> Git already supports a branching workflow, this would allow us to branch
> off master in git and spin up a registry pointed at the branch.  We could
> also use pull requests to update the master registry (currently that would
> require bouncing the maste but that could be improved as well)
>
> Further out we could also store pointers to binaries in git and then have a
> blob store for accessing them, (S3, nfs, sftp, whatever) similarly to how
> git-lfs [2] works so that it would be possible to use git workflows for
> managing branching of extensions as well.
>
> Note: I'm not suggesting forcing git on all registry users, just having it
> as a configuration option :)
>
> [1] https://github.com/Chaffelson/nipyapi
> [2] https://git-lfs.github.com/
>
> Thanks,
> Bryan

Mime
View raw message