In short, not at the moment, but there are plans to facilitate this process much in the way as you described.

I think you would be greatly aided by a concept that Joe presented for discussion to the community.  This is only in design, but there are many people that are mapping components of their flows to this variety of flow with these functional groups.  

The email subject was titled: "[DISCUSS] Feature proposal: 'Function Groups' and wormholes"

Initial write-ups are available for each of these concepts at:

Wormhole writeup:

Function Group writeup:

We would love for any additional perspectives you can provide in solving your use cases.


On Mon, Aug 10, 2015 at 11:17 AM, John Titus <> wrote:
Sorry for the long email, I couldn't figure out a shorter way to explain the context.

I have several processor groups in our production system. While they all differ as to where they get the data, towards the end of each processor group they look very similar; that is, they all store the data they've retrieved in a similar manner. However, at the very end of each processor group I need to do some cleanup, which differs by processor group.

In order to "DRY" up my processor groups, I created an external, "Store File" processor group. I modified the existing processor group, "A", to use it. When "Store File" finishes, it routes to an Output Port called "Storage Done". Back in "A" I have an input port that grabs that and does the cleanup.  Using "A" hooked into "Storage File" and then back to "A" for cleanup worked great.

Then I tried to hook in processor group "B" to "Store File".  It also worked, until "Store File" sends the data to the "Storage Done" output port.  Because this output port for "Store File" is now hooked to the cleanup input ports in "A" and "B", flow files are sent to both A & B. This causes errors because the cleanup is different for both groups.

I guess I ran into this problem because, in my mental model, I was thinking of Input/Output ports like calling a function - when the function was done, it would return to the calling function (the calling processor group). Obviously, I was wrong about that.

My current planned solution is to add an attribute to both "A" and "B" (and C, D, etc) that holds the Processor Group name, and then inside of "Store File" use the RouteOnAttribute to route to the "correct" output port.  However, this isn't ideal - "Store File" is now somewhat "aware" of the other processor groups and thus more strongly coupled than I'd like.

Is there a better way to DRY up code in NiFi than what I'm doing?