nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aldrin Piri <aldrinp...@gmail.com>
Subject Re: Output Port Routing and DRYing up Nifi processor Groups
Date Mon, 10 Aug 2015 15:22:53 GMT
John,

In short, not at the moment, but there are plans to facilitate this process
much in the way as you described.

I think you would be greatly aided by a concept that Joe presented for
discussion to the community.  This is only in design, but there are many
people that are mapping components of their flows to this variety of flow
with these functional groups.

The email subject was titled: "[DISCUSS] Feature proposal: 'Function
Groups' and wormholes"

Initial write-ups are available for each of these concepts at:

Wormhole writeup:
https://cwiki.apache.org/confluence/display/NIFI/Wormhole+Connections

Function Group writeup:
https://cwiki.apache.org/confluence/display/NIFI/Reference-able+Process+Groups

We would love for any additional perspectives you can provide in solving
your use cases.

Thanks!

On Mon, Aug 10, 2015 at 11:17 AM, John Titus <john.titus@trustedconcepts.com
> wrote:

> Sorry for the long email, I couldn't figure out a shorter way to explain
> the context.
>
> I have several processor groups in our production system. While they all
> differ as to where they get the data, towards the end of each processor
> group they look very similar; that is, they all store the data they've
> retrieved in a similar manner. However, at the very end of each processor
> group I need to do some cleanup, which differs by processor group.
>
> In order to "DRY" up my processor groups, I created an external, "Store
> File" processor group. I modified the existing processor group, "A", to use
> it. When "Store File" finishes, it routes to an Output Port called "Storage
> Done". Back in "A" I have an input port that grabs that and does the
> cleanup.  Using "A" hooked into "Storage File" and then back to "A" for
> cleanup worked great.
>
> Then I tried to hook in processor group "B" to "Store File".  It also
> worked, until "Store File" sends the data to the "Storage Done" output
> port.  Because this output port for "Store File" is now hooked to the
> cleanup input ports in "A" and "B", flow files are sent to both A & B. This
> causes errors because the cleanup is different for both groups.
>
> I guess I ran into this problem because, in my mental model, I was
> thinking of Input/Output ports like calling a function - when the function
> was done, it would return to the calling function (the calling processor
> group). Obviously, I was wrong about that.
>
> My current planned solution is to add an attribute to both "A" and "B"
> (and C, D, etc) that holds the Processor Group name, and then inside of
> "Store File" use the RouteOnAttribute to route to the "correct" output
> port.  However, this isn't ideal - "Store File" is now somewhat "aware" of
> the other processor groups and thus more strongly coupled than I'd like.
>
> Is there a better way to DRY up code in NiFi than what I'm doing?
>
> John
>
>
>

Mime
View raw message