nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Aslan <scottyas...@gmail.com>
Subject Re: [DISCUSS] MiNiFi Flow Designer
Date Thu, 01 Mar 2018 19:21:35 GMT
AndyC/Aldrin,

The UI/UX design system sub project currently under discussion will be
based off of the work completed during the NiFi Registry and will not
contain any UI/UX functionality from NiFi. The long term goal would be to
upgrade angular in the NiFi UI in order to leverage this new design system.

-Scotty


On Thu, Mar 1, 2018 at 1:45 PM, Kevin Doran <kdoran@apache.org> wrote:

> Aldrin,
>
> Thanks a lot for taking the time to write up such a detailed and well
> thought out proposal.
>
> I really like the idea of publishing flows to NiFi Registry that then get
> deployed to MiNiFi agents via the C2 Server. I don't have much to add
> there, and I think that you've identified one of the big missing parts to
> make that experience user-friendly -- how to design and author the flows in
> the first place.
>
> The MiNiFi Flow Designer would fill that need in a really elegant way with
> a low learning curve for NiFi users. Having the ability to design and
> author flows in an environment other than the "runtime" environment one is
> targeting would be a huge step forward. Not just for MiNiFi, but
> multi-environment deployments of NiFi, where one might be developing a flow
> in an environment other than where they intend to run it. This would make
> the SDLC options for NiFi flows even more flexible, which is a big part of
> the goal of the new NiFi Registry.
>
> It certainly presents some challenges, but I definitely think the upside
> for MiNiFi flow authoring would be huge.
>
> Thanks,
> Kevin
>
> On 2/28/18, 07:53, "Aldrin Piri" <aldrinpiri@gmail.com> wrote:
>
>     It appears I accidentally left off the actual links.  Here they are in
> all
>     their glory:
>
>     [1]
>     https://cwiki.apache.org/confluence/display/MINIFI/
> MiNiFi+Command+and+Control
>     [2]
>     https://github.com/apache/nifi-minifi/blob/master/
> minifi-docs/src/main/markdown/System_Admin_Guide.md#config-file
>     [3] https://issues.apache.org/jira/browse/MINIFICPP-36
>     [4] https://github.com/apache/nifi-minifi/tree/master/minifi-c2
>     [5]
>     https://github.com/apache/nifi-minifi/tree/master/
> minifi-c2#configuration-providers
>     [6]
>     https://github.com/apache/nifi-minifi/blob/master/
> minifi-docs/src/main/markdown/System_Admin_Guide.md#
> automatic-warm-redeploy
>     [7]
>     https://cwiki.apache.org/confluence/display/NIFI/
> Extension+Repositories+%28aka+Extension+Registry%29+for+
> Dynamically-loaded+Extensions
>     [8]
>     https://cwiki.apache.org/confluence/display/MINIFI/
> MiNiFi+Command+and+Control#MiNiFiCommandandControl-FlowAuthorshipDetails
>
>
>     On Wed, Feb 28, 2018 at 1:53 AM, Aldrin Piri <aldrinpiri@gmail.com>
> wrote:
>
>     > Hey folks,
>     >
>     > With the release of Registry, I’ve been contemplating leveraging it
> and
>     > the current codebase of NiFi to facilitate ease of use for MiNiFi
> flow
>     > design.  One of the areas I would like to form a concerted effort
> around is
>     > that of the Command and Control (C2) functionality originally
> presented
>     > when the MiNiFi effort was started and further expounded upon with a
>     > feature proposal [1].  In that proposal, while the names are dated,
> we have
>     > components that fulfill some of the core building blocks toward the
> overall
>     > vision of C2.
>     >
>     > For those not intimately familiar, MiNiFi is primarily configured
> via a
>     > YAML configuration file.  [2]  The initial idea was that flows would
> be
>     > simple a source, maybe a transform or two, and then transmission.
> The
>     > notion was that hand creation of one of these files would not be
> overly
>     > onerous.  It became apparent from questions, however, that users
> were doing
>     > more complex flows and this YAML ranged from tedious to difficult
> for those
>     > situations.  This precipitated the best approach currently available,
>     > utilizing a NiFi instance as a way to generate a template.  This
> template
>     > was then exported, and using our provided toolkit to get YAML out
> that is
>     > deployable with your instance.  Better, but far from ideal.
>     >
>     > A couple approaches came to mind to further narrow this gap of user
>     > expectation and realities of our configuration.
>     >
>     > (1) In our current state, we have heavily relied on convenience of
>     > component overlap.  Generally speaking, we relied on the notion that
> NiFi’s
>     > bundled components are a superset of those found in MiNiFi
>     > implementations.  We could have continued that, but this would not
>     > necessarily free the user of the file handling and transformation
>     > processes.  Additionally, this comes with caveats, among them that
> given
>     > the ways in which C++ allows us different accesses, there are some
>     > interesting processors that work better for that more native
> approach and
>     > might not ever justify having an equivalent in Java.
>     >
>     > (2) Providing a first class experience in NiFi for MiNiFi flow
> designing
>     > making use of the existing UI components as a base and a new design
> mode
>     > specifically for providing a clean user experience, letting users
> operate
>     > in a different context but a familiar environment.  Certainly
> preferred,
>     > but how this fit was not clear.
>     >
>     > As a crash course in MiNiFi efforts, there have been some key
>     > contributions that I believe when more strongly united can generate
> a very
>     > complete authoring and flow update process.  Some of these items are:
>     >
>     >    - C2 API Protocol specification and implementation in C++ [3]
>     >    - C2 Server with the notion of serving different flows [4] backed
> by
>     >    varying implementations, one of which was via REST to access NiFi
> Templates
>     >    directly [5]
>     >    - Warm Redeploy in MiNiFi Java [6]
>     >
>     >
>     > One of the first questions to come to mind was, “Is within NiFi the
> right
>     > place?”
>     >
>     > I think there are several reasons that this is reasonable. Seeing
> some of
>     > the discussion around the consolidation of common UI components
> presented
>     > by Scott Aslan, I think sharing the core of the UI elements between
> the
>     > traditional NiFi and MiNiFi flow designs is a logical extension.
> While
>     > those flows in MiNiFi are not real-time changes we are dealing with
> the
>     > same design elements in creating directed graphs.  Additionally,
> while
>     > MiNiFi instances will likely not ever have the same breadth of
> extensions
>     > supported, there is nothing inherently prohibitive of a more complex
> graph
>     > comprised of larger numbers of edges and/or vertices.  In my view,
> the
>     > mechanics are the same but just being performed in a different
> contexts.
>     > In the case of MiNiFi, this is effectively an “offline” context.  By
>     > "offline," I mean we are not strictly bound to loaded components
> such as
>     > that in the current NiFi context consisting of items from unpacked
> NARs.
>     >
>     > How can we minimize the impact to NiFi?
>     >
>     > Given that this is a new capability, I don’t know that we
> necessarily need
>     > to interfere with the existing APIs and could possibly also make
> this an
>     > optional feature.  From this standpoint, we could develop new
> endpoints for
>     > this specific context.  Some of this work would rely on external
> services
>     > and/or definitions to drive the different context.  For example,
> without
>     > NARs acting as the source of available components, something like
> the C2
>     > server could provide extensions for MiNiFi instances it is aware of
> to help
>     > drive the experience.  This data would be a skeletal view of a
> component
>     > definition/interface without any of the inherent code backing it
> needed.
>     >
>     > How do we maximize the utility of these efforts?
>     >
>     > With the idea about powering MiNiFi flow design with specific sets of
>     > components, I could see this extending to even NiFi itself.
> Consider the
>     > use case where something like I wish to make use of additional
> artifacts
>     > along the lines of [7] and an extension of the current Registry
>     > implementation.  As a simple example, with an understanding of what
> the
>     > universe of components is available to my system and not just those
>     > currently loaded, it would be possible for users to design for other
>     > instances.  With these hooks in place, there are a few permutations
> of this
>     > relative concept that enable us to carry out some of the “nice to
> haves” we
>     > haven’t quite been able to address.
>     >
>     > How does this all come together for MiNiFi?
>     >
>     > I have created a work of sheer artistic mastery on the Command and
> Control
>     > proposal to illustrate a rough sequence and architecture. [8]  This
> diagram
>     > shows from entering MiNiFi flow design through to an instance
> updating to
>     > the new flow.
>     >
>     >
>     > Overall, Registry was a tremendous step forward for the NiFi
> ecosystem.  I
>     > think in our current state, we have reached a convergence of
> capabilities
>     > that allow us to build higher level abstractions for dataflow
> management.
>     > I think the approach of (2) is a great first foray into that realm
> that
>     > provides an immediate solution for a gap in our MiNiFi efforts.
>     > Additionally, this approach lays the foundation for additional
> applications
>     > possibly inclusive of core NiFi.
>     >
>     > I look forward to your thoughts and commentary.  Certainly many
> details to
>     > think and reason about, but I see some incredible potential with the
> work
>     > we've done thus far both near and long term.
>     >
>     > Thanks for your time!
>     > --aldrin
>     >
>     >
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message