nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Titus <>
Subject Output Port Routing and DRYing up Nifi processor Groups
Date Mon, 10 Aug 2015 15:17:20 GMT
Sorry for the long email, I couldn't figure out a shorter way to explain the context.

I have several processor groups in our production system. While they all differ as to where
they get the data, towards the end of each processor group they look very similar; that is,
they all store the data they've retrieved in a similar manner. However, at the very end of
each processor group I need to do some cleanup, which differs by processor group.

In order to "DRY" up my processor groups, I created an external, "Store File" processor group.
I modified the existing processor group, "A", to use it. When "Store File" finishes, it routes
to an Output Port called "Storage Done". Back in "A" I have an input port that grabs that
and does the cleanup.  Using "A" hooked into "Storage File" and then back to "A" for cleanup
worked great.

Then I tried to hook in processor group "B" to "Store File".  It also worked, until "Store
File" sends the data to the "Storage Done" output port.  Because this output port for "Store
File" is now hooked to the cleanup input ports in "A" and "B", flow files are sent to both
A & B. This causes errors because the cleanup is different for both groups.

I guess I ran into this problem because, in my mental model, I was thinking of Input/Output
ports like calling a function - when the function was done, it would return to the calling
function (the calling processor group). Obviously, I was wrong about that.

My current planned solution is to add an attribute to both "A" and "B" (and C, D, etc) that
holds the Processor Group name, and then inside of "Store File" use the RouteOnAttribute to
route to the "correct" output port.  However, this isn't ideal - "Store File" is now somewhat
"aware" of the other processor groups and thus more strongly coupled than I'd like.

Is there a better way to DRY up code in NiFi than what I'm doing?


View raw message