nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aldrin Piri <aldrinp...@gmail.com>
Subject Re: [DISCUSS] MiNiFi Flow Designer
Date Fri, 02 Mar 2018 03:12:31 GMT
Hey folks,

Thanks for all the input and feedback thus far.  Going to try to address
some points that came up that I have some thoughts on.

Andy,

As for continued YAML support, it is the intent to do so moving forward.
In some respects, at least for initial flow authoring/deployment
capabilities, I think it is a fundamental part where the C2 server would
help to translate the Versioned Flow content into a YAML format.  As we get
more sophisticated we could have other, more efficient/compact
representations in addition.

Scotty,

In terms of the UI/UX work with the subproject being discussed, it was less
about using those components directly and more of an analog where we are
consolidating efforts into one place.  The comparison I was trying to make
was that whatever the UI/UX is, new or old, has some very strong
similarities between both NiFi and MiNiFi with slightly different
contexts.  Apologies for the lack of clarity on that front.

Andrew,

Great thoughts and I think your concept of the groups with the available
extensions is key.  With such hooks in place, the applicability of the
general workflow and tooling could have some broader applications to other
efforts as has been suggested by some of the other folks in this chain.
Knowing that components exist powered by something like previously
discussed Extension handling in Registry, even if they are not currently on
"this" instance, can open up some really interesting possibilities in terms
of generating assemblies and other party tricks.

Thanks for your thoughts and time!

--aldrin

On Thu, Mar 1, 2018 at 8:12 PM, Andy LoPresto <alopresto@apache.org> wrote:

> I wonder if extending this model to all *NiFi flows makes sense. A common
> user request we have seen is to provide for a “flow development”
> environment separate from the “flow execution” environment (this was one of
> the leading impetuses for the FDLC enhancement effort that led to the
> Registry). Some users expect to allow for a priori flow development, which
> is then promoted as a static flow definition to a disparate production-like
> environment where it can run and be monitored but not be editable.
>
> I think extending the NiFi Flow Designer + MiNiFi “flow execution”
> scenario to n-many execution environments (MiNiFi - Java, MiNiFi - C++,
> NiFi “headless”/“run-only”) might make sense here. We already have code to
> export the same flow (object graph) to YAML, XML, etc., so the “Design”
> portion could be on a specific server/network/etc. and have specific user
> access and permissions, and could be connected to a Registry instance, so
> that when a flow is deemed “complete” it is exported/saved/checked-in to
> the Registry, and then one or more “execution environments” (MiNiFi - *,
> a regular NiFi node, a “monitor-only” NiFi node) could all check-out the
> flow in the appropriate format and import/run that flow.
>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <alopresto.apache@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Mar 1, 2018, at 4:39 PM, Andrew Grande <aperepel@gmail.com> wrote:
>
> My 2c.
>
> Using NiFi registry for both NiFi and MiNiFi flows is ideal. For the
> designer, IMO, adding another process/service to host it is not too great,
> as it adds to the complexity of overall ecosystem (the nifi/minifi java/cpp
> is already quite involved with c2 server, etc). I would like to see a
> familiar NiFi UI concept evolve to have a 'profile' e.g. on a group level.
> This profile would dictate which processors are available for that
> particular runtime. The same profile eould be associated with a flow in the
> registry.
>
> The main concept is to stay within the same familiar design environment for
> all deployment targets, but make NiFi understand what set of available
> components and behaviors are available for the current target at hand.
>
> Andrew
>
> On Thu, Mar 1, 2018, 7:30 PM Jeff Zemerick <jzemerick@apache.org> wrote:
>
> I think it sounds like a good idea. The majority of the MiNiFi flows I have
> made are all fairly simple and I have not found the YAML authoring to be
> overly burdensome. The more complex flows were done in NiFi and converted.
> So as long as the option to make the flows by hand remains (and I don't
> think there was anything in the proposal to the contrary), I think it would
> be a beneficial addition.
>
> Jeff
>
>
> On Thu, Mar 1, 2018 at 2:21 PM, Scott Aslan <scottyaslan@gmail.com> wrote:
>
> AndyC/Aldrin,
>
> The UI/UX design system sub project currently under discussion will be
> based off of the work completed during the NiFi Registry and will not
> contain any UI/UX functionality from NiFi. The long term goal would be to
> upgrade angular in the NiFi UI in order to leverage this new design
>
> system.
>
>
> -Scotty
>
>
> On Thu, Mar 1, 2018 at 1:45 PM, Kevin Doran <kdoran@apache.org> wrote:
>
> Aldrin,
>
> Thanks a lot for taking the time to write up such a detailed and well
> thought out proposal.
>
> I really like the idea of publishing flows to NiFi Registry that then
>
> get
>
> deployed to MiNiFi agents via the C2 Server. I don't have much to add
> there, and I think that you've identified one of the big missing parts
>
> to
>
> make that experience user-friendly -- how to design and author the
>
> flows
>
> in
>
> the first place.
>
> The MiNiFi Flow Designer would fill that need in a really elegant way
>
> with
>
> a low learning curve for NiFi users. Having the ability to design and
> author flows in an environment other than the "runtime" environment one
>
> is
>
> targeting would be a huge step forward. Not just for MiNiFi, but
> multi-environment deployments of NiFi, where one might be developing a
>
> flow
>
> in an environment other than where they intend to run it. This would
>
> make
>
> the SDLC options for NiFi flows even more flexible, which is a big part
>
> of
>
> the goal of the new NiFi Registry.
>
> It certainly presents some challenges, but I definitely think the
>
> upside
>
> for MiNiFi flow authoring would be huge.
>
> Thanks,
> Kevin
>
> On 2/28/18, 07:53, "Aldrin Piri" <aldrinpiri@gmail.com> wrote:
>
>    It appears I accidentally left off the actual links.  Here they are
>
> in
>
> all
>    their glory:
>
>    [1]
>    https://cwiki.apache.org/confluence/display/MINIFI/
> MiNiFi+Command+and+Control
>    [2]
>    https://github.com/apache/nifi-minifi/blob/master/
> minifi-docs/src/main/markdown/System_Admin_Guide.md#config-file
>    [3] https://issues.apache.org/jira/browse/MINIFICPP-36
>    [4] https://github.com/apache/nifi-minifi/tree/master/minifi-c2
>    [5]
>    https://github.com/apache/nifi-minifi/tree/master/
> minifi-c2#configuration-providers
>    [6]
>    https://github.com/apache/nifi-minifi/blob/master/
> minifi-docs/src/main/markdown/System_Admin_Guide.md#
> automatic-warm-redeploy
>    [7]
>    https://cwiki.apache.org/confluence/display/NIFI/
> Extension+Repositories+%28aka+Extension+Registry%29+for+
> Dynamically-loaded+Extensions
>    [8]
>    https://cwiki.apache.org/confluence/display/MINIFI/
>
> MiNiFi+Command+and+Control#MiNiFiCommandandControl-FlowAuthorshipDetails
>
>
>
>    On Wed, Feb 28, 2018 at 1:53 AM, Aldrin Piri <aldrinpiri@gmail.com
>
>
> wrote:
>
> Hey folks,
>
> With the release of Registry, I’ve been contemplating leveraging
>
> it
>
> and
>
> the current codebase of NiFi to facilitate ease of use for MiNiFi
>
> flow
>
> design.  One of the areas I would like to form a concerted effort
>
> around is
>
> that of the Command and Control (C2) functionality originally
>
> presented
>
> when the MiNiFi effort was started and further expounded upon
>
> with
>
> a
>
> feature proposal [1].  In that proposal, while the names are
>
> dated,
>
> we have
>
> components that fulfill some of the core building blocks toward
>
> the
>
> overall
>
> vision of C2.
>
> For those not intimately familiar, MiNiFi is primarily configured
>
> via a
>
> YAML configuration file.  [2]  The initial idea was that flows
>
> would
>
> be
>
> simple a source, maybe a transform or two, and then transmission.
>
> The
>
> notion was that hand creation of one of these files would not be
>
> overly
>
> onerous.  It became apparent from questions, however, that users
>
> were doing
>
> more complex flows and this YAML ranged from tedious to difficult
>
> for those
>
> situations.  This precipitated the best approach currently
>
> available,
>
> utilizing a NiFi instance as a way to generate a template.  This
>
> template
>
> was then exported, and using our provided toolkit to get YAML out
>
> that is
>
> deployable with your instance.  Better, but far from ideal.
>
> A couple approaches came to mind to further narrow this gap of
>
> user
>
> expectation and realities of our configuration.
>
> (1) In our current state, we have heavily relied on convenience
>
> of
>
> component overlap.  Generally speaking, we relied on the notion
>
> that
>
> NiFi’s
>
> bundled components are a superset of those found in MiNiFi
> implementations.  We could have continued that, but this would
>
> not
>
> necessarily free the user of the file handling and transformation
> processes.  Additionally, this comes with caveats, among them
>
> that
>
> given
>
> the ways in which C++ allows us different accesses, there are
>
> some
>
> interesting processors that work better for that more native
>
> approach and
>
> might not ever justify having an equivalent in Java.
>
> (2) Providing a first class experience in NiFi for MiNiFi flow
>
> designing
>
> making use of the existing UI components as a base and a new
>
> design
>
> mode
>
> specifically for providing a clean user experience, letting users
>
> operate
>
> in a different context but a familiar environment.  Certainly
>
> preferred,
>
> but how this fit was not clear.
>
> As a crash course in MiNiFi efforts, there have been some key
> contributions that I believe when more strongly united can
>
> generate
>
> a very
>
> complete authoring and flow update process.  Some of these items
>
> are:
>
>
>   - C2 API Protocol specification and implementation in C++ [3]
>   - C2 Server with the notion of serving different flows [4]
>
> backed
>
> by
>
>   varying implementations, one of which was via REST to access
>
> NiFi
>
> Templates
>
>   directly [5]
>   - Warm Redeploy in MiNiFi Java [6]
>
>
> One of the first questions to come to mind was, “Is within NiFi
>
> the
>
> right
>
> place?”
>
> I think there are several reasons that this is reasonable. Seeing
>
> some of
>
> the discussion around the consolidation of common UI components
>
> presented
>
> by Scott Aslan, I think sharing the core of the UI elements
>
> between
>
> the
>
> traditional NiFi and MiNiFi flow designs is a logical extension.
>
> While
>
> those flows in MiNiFi are not real-time changes we are dealing
>
> with
>
> the
>
> same design elements in creating directed graphs.  Additionally,
>
> while
>
> MiNiFi instances will likely not ever have the same breadth of
>
> extensions
>
> supported, there is nothing inherently prohibitive of a more
>
> complex
>
> graph
>
> comprised of larger numbers of edges and/or vertices.  In my
>
> view,
>
> the
>
> mechanics are the same but just being performed in a different
>
> contexts.
>
> In the case of MiNiFi, this is effectively an “offline” context.
>
> By
>
> "offline," I mean we are not strictly bound to loaded components
>
> such as
>
> that in the current NiFi context consisting of items from
>
> unpacked
>
> NARs.
>
>
> How can we minimize the impact to NiFi?
>
> Given that this is a new capability, I don’t know that we
>
> necessarily need
>
> to interfere with the existing APIs and could possibly also make
>
> this an
>
> optional feature.  From this standpoint, we could develop new
>
> endpoints for
>
> this specific context.  Some of this work would rely on external
>
> services
>
> and/or definitions to drive the different context.  For example,
>
> without
>
> NARs acting as the source of available components, something like
>
> the C2
>
> server could provide extensions for MiNiFi instances it is aware
>
> of
>
> to help
>
> drive the experience.  This data would be a skeletal view of a
>
> component
>
> definition/interface without any of the inherent code backing it
>
> needed.
>
>
> How do we maximize the utility of these efforts?
>
> With the idea about powering MiNiFi flow design with specific
>
> sets
>
> of
>
> components, I could see this extending to even NiFi itself.
>
> Consider the
>
> use case where something like I wish to make use of additional
>
> artifacts
>
> along the lines of [7] and an extension of the current Registry
> implementation.  As a simple example, with an understanding of
>
> what
>
> the
>
> universe of components is available to my system and not just
>
> those
>
> currently loaded, it would be possible for users to design for
>
> other
>
> instances.  With these hooks in place, there are a few
>
> permutations
>
> of this
>
> relative concept that enable us to carry out some of the “nice to
>
> haves” we
>
> haven’t quite been able to address.
>
> How does this all come together for MiNiFi?
>
> I have created a work of sheer artistic mastery on the Command
>
> and
>
> Control
>
> proposal to illustrate a rough sequence and architecture. [8]
>
> This
>
> diagram
>
> shows from entering MiNiFi flow design through to an instance
>
> updating to
>
> the new flow.
>
>
> Overall, Registry was a tremendous step forward for the NiFi
>
> ecosystem.  I
>
> think in our current state, we have reached a convergence of
>
> capabilities
>
> that allow us to build higher level abstractions for dataflow
>
> management.
>
> I think the approach of (2) is a great first foray into that
>
> realm
>
> that
>
> provides an immediate solution for a gap in our MiNiFi efforts.
> Additionally, this approach lays the foundation for additional
>
> applications
>
> possibly inclusive of core NiFi.
>
> I look forward to your thoughts and commentary.  Certainly many
>
> details to
>
> think and reason about, but I see some incredible potential with
>
> the
>
> work
>
> we've done thus far both near and long term.
>
> Thanks for your time!
> --aldrin
>
>
>
>
>
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message