nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Doran <kdo...@apache.org>
Subject Re: [DISCUSS] MiNiFi Flow Designer
Date Thu, 01 Mar 2018 18:45:26 GMT
Aldrin,

Thanks a lot for taking the time to write up such a detailed and well thought out proposal.

I really like the idea of publishing flows to NiFi Registry that then get deployed to MiNiFi
agents via the C2 Server. I don't have much to add there, and I think that you've identified
one of the big missing parts to make that experience user-friendly -- how to design and author
the flows in the first place.

The MiNiFi Flow Designer would fill that need in a really elegant way with a low learning
curve for NiFi users. Having the ability to design and author flows in an environment other
than the "runtime" environment one is targeting would be a huge step forward. Not just for
MiNiFi, but multi-environment deployments of NiFi, where one might be developing a flow in
an environment other than where they intend to run it. This would make the SDLC options for
NiFi flows even more flexible, which is a big part of the goal of the new NiFi Registry.

It certainly presents some challenges, but I definitely think the upside for MiNiFi flow authoring
would be huge.

Thanks,
Kevin

On 2/28/18, 07:53, "Aldrin Piri" <aldrinpiri@gmail.com> wrote:

    It appears I accidentally left off the actual links.  Here they are in all
    their glory:
    
    [1]
    https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control
    [2]
    https://github.com/apache/nifi-minifi/blob/master/minifi-docs/src/main/markdown/System_Admin_Guide.md#config-file
    [3] https://issues.apache.org/jira/browse/MINIFICPP-36
    [4] https://github.com/apache/nifi-minifi/tree/master/minifi-c2
    [5]
    https://github.com/apache/nifi-minifi/tree/master/minifi-c2#configuration-providers
    [6]
    https://github.com/apache/nifi-minifi/blob/master/minifi-docs/src/main/markdown/System_Admin_Guide.md#automatic-warm-redeploy
    [7]
    https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+%28aka+Extension+Registry%29+for+Dynamically-loaded+Extensions
    [8]
    https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control#MiNiFiCommandandControl-FlowAuthorshipDetails
    
    
    On Wed, Feb 28, 2018 at 1:53 AM, Aldrin Piri <aldrinpiri@gmail.com> wrote:
    
    > Hey folks,
    >
    > With the release of Registry, I’ve been contemplating leveraging it and
    > the current codebase of NiFi to facilitate ease of use for MiNiFi flow
    > design.  One of the areas I would like to form a concerted effort around is
    > that of the Command and Control (C2) functionality originally presented
    > when the MiNiFi effort was started and further expounded upon with a
    > feature proposal [1].  In that proposal, while the names are dated, we have
    > components that fulfill some of the core building blocks toward the overall
    > vision of C2.
    >
    > For those not intimately familiar, MiNiFi is primarily configured via a
    > YAML configuration file.  [2]  The initial idea was that flows would be
    > simple a source, maybe a transform or two, and then transmission.  The
    > notion was that hand creation of one of these files would not be overly
    > onerous.  It became apparent from questions, however, that users were doing
    > more complex flows and this YAML ranged from tedious to difficult for those
    > situations.  This precipitated the best approach currently available,
    > utilizing a NiFi instance as a way to generate a template.  This template
    > was then exported, and using our provided toolkit to get YAML out that is
    > deployable with your instance.  Better, but far from ideal.
    >
    > A couple approaches came to mind to further narrow this gap of user
    > expectation and realities of our configuration.
    >
    > (1) In our current state, we have heavily relied on convenience of
    > component overlap.  Generally speaking, we relied on the notion that NiFi’s
    > bundled components are a superset of those found in MiNiFi
    > implementations.  We could have continued that, but this would not
    > necessarily free the user of the file handling and transformation
    > processes.  Additionally, this comes with caveats, among them that given
    > the ways in which C++ allows us different accesses, there are some
    > interesting processors that work better for that more native approach and
    > might not ever justify having an equivalent in Java.
    >
    > (2) Providing a first class experience in NiFi for MiNiFi flow designing
    > making use of the existing UI components as a base and a new design mode
    > specifically for providing a clean user experience, letting users operate
    > in a different context but a familiar environment.  Certainly preferred,
    > but how this fit was not clear.
    >
    > As a crash course in MiNiFi efforts, there have been some key
    > contributions that I believe when more strongly united can generate a very
    > complete authoring and flow update process.  Some of these items are:
    >
    >    - C2 API Protocol specification and implementation in C++ [3]
    >    - C2 Server with the notion of serving different flows [4] backed by
    >    varying implementations, one of which was via REST to access NiFi Templates
    >    directly [5]
    >    - Warm Redeploy in MiNiFi Java [6]
    >
    >
    > One of the first questions to come to mind was, “Is within NiFi the right
    > place?”
    >
    > I think there are several reasons that this is reasonable. Seeing some of
    > the discussion around the consolidation of common UI components presented
    > by Scott Aslan, I think sharing the core of the UI elements between the
    > traditional NiFi and MiNiFi flow designs is a logical extension.  While
    > those flows in MiNiFi are not real-time changes we are dealing with the
    > same design elements in creating directed graphs.  Additionally, while
    > MiNiFi instances will likely not ever have the same breadth of extensions
    > supported, there is nothing inherently prohibitive of a more complex graph
    > comprised of larger numbers of edges and/or vertices.  In my view, the
    > mechanics are the same but just being performed in a different contexts.
    > In the case of MiNiFi, this is effectively an “offline” context.  By
    > "offline," I mean we are not strictly bound to loaded components such as
    > that in the current NiFi context consisting of items from unpacked NARs.
    >
    > How can we minimize the impact to NiFi?
    >
    > Given that this is a new capability, I don’t know that we necessarily need
    > to interfere with the existing APIs and could possibly also make this an
    > optional feature.  From this standpoint, we could develop new endpoints for
    > this specific context.  Some of this work would rely on external services
    > and/or definitions to drive the different context.  For example, without
    > NARs acting as the source of available components, something like the C2
    > server could provide extensions for MiNiFi instances it is aware of to help
    > drive the experience.  This data would be a skeletal view of a component
    > definition/interface without any of the inherent code backing it needed.
    >
    > How do we maximize the utility of these efforts?
    >
    > With the idea about powering MiNiFi flow design with specific sets of
    > components, I could see this extending to even NiFi itself.  Consider the
    > use case where something like I wish to make use of additional artifacts
    > along the lines of [7] and an extension of the current Registry
    > implementation.  As a simple example, with an understanding of what the
    > universe of components is available to my system and not just those
    > currently loaded, it would be possible for users to design for other
    > instances.  With these hooks in place, there are a few permutations of this
    > relative concept that enable us to carry out some of the “nice to haves” we
    > haven’t quite been able to address.
    >
    > How does this all come together for MiNiFi?
    >
    > I have created a work of sheer artistic mastery on the Command and Control
    > proposal to illustrate a rough sequence and architecture. [8]  This diagram
    > shows from entering MiNiFi flow design through to an instance updating to
    > the new flow.
    >
    >
    > Overall, Registry was a tremendous step forward for the NiFi ecosystem.  I
    > think in our current state, we have reached a convergence of capabilities
    > that allow us to build higher level abstractions for dataflow management.
    > I think the approach of (2) is a great first foray into that realm that
    > provides an immediate solution for a gap in our MiNiFi efforts.
    > Additionally, this approach lays the foundation for additional applications
    > possibly inclusive of core NiFi.
    >
    > I look forward to your thoughts and commentary.  Certainly many details to
    > think and reason about, but I see some incredible potential with the work
    > we've done thus far both near and long term.
    >
    > Thanks for your time!
    > --aldrin
    >
    >
    



Mime
View raw message