nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Rosander <bryanrosan...@gmail.com>
Subject NiFi Registry feature branch workflow (possibly via Git-Backed MetadataService, FlowPersistenceProvider)
Date Tue, 05 Mar 2019 16:02:05 GMT
Hi all,

We're trying to implement a feature branch workflow with NiFi Registry.
The basic idea is that you'd be able to have one or more branches off of
master and then the capability to merge changes back in when they're ready.

Our flow has many versioned process groups that are nested several layers
deep.

There are a couple of approaches that we're thinking about:

NiPyApi [1] (hacky but should work without registry changes):
1. Keep track of versions when branch was created
2. Use NiPyApi to compile a list of changed flows since the branch
3. Apply changes to master registry from the bottom process group(s) up,
updating version numbers of child pgs as we go, writing out to some sort of
patch file
4. Verify patch file looks right, use NiPyApi to apply it (could do peer
review as part of this)

Git-Based:
Create a git-backed MetadataService that can coexist with
the GitFlowPersistenceProvider (share a repository, not mess with each
others' metadata)

Git already supports a branching workflow, this would allow us to branch
off master in git and spin up a registry pointed at the branch.  We could
also use pull requests to update the master registry (currently that would
require bouncing the maste but that could be improved as well)

Further out we could also store pointers to binaries in git and then have a
blob store for accessing them, (S3, nfs, sftp, whatever) similarly to how
git-lfs [2] works so that it would be possible to use git workflows for
managing branching of extensions as well.

Note: I'm not suggesting forcing git on all registry users, just having it
as a configuration option :)

[1] https://github.com/Chaffelson/nipyapi
[2] https://git-lfs.github.com/

Thanks,
Bryan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message