nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lee Laim (leelaim)" <>
Subject RE: [EXT] Back Pressure on Process Group
Date Thu, 11 Oct 2018 19:49:31 GMT
Hi John,  

You can send the initiating flowfile into the Process Group and simultaneously send a duplicate
flowfile around the process group into a MergeContent processor.  This duplicate will act
as a back pressure "latch".

Upon successful exit of the process group, Merge Content with a strict correlation strategy
will clear the back pressure latch, allowing the next flowfile into the group.

Personally, I think Wait/Notify is the more elegant solution, but have successfully used the
back pressure latch before Wait/Notify was readily available.   

This thread might offer some additional insight:


-----Original Message-----
From: John McGinn [] 
Sent: Thursday, October 11, 2018 12:17 PM
Subject: [EXT] Back Pressure on Process Group

I've been going through mailing list archives, and looking at blog posts around Wait/Notify,
and these don't seem to be the solution for my use case.

My basic use case is as follows. I have 4 DB tables, 3 of which are id/name pairs (office
name, city, state), and the 4th table joins the 3 ids together to a new id which is used elsewhere
in the database.

Using NiFi to injest data from a different database system, we have to verify if that office
is active, and if it isn't active or non-existent, create a new record, as well as any of
the other 3 tables necessary.

The first step, then, is to join the 4 tables together to search for the name fields, and
if the join comes back with a row, use that top level id as an attribute. No problem, works
fine. (FetchDatabaseTable -> AvroToJson -> EvaluateJsonPath, etc.) If the join comes
back empty, I need to insert rows for the 3 pieces and then join them together. Ideally, this
would be a flow of 3 PutSqls, then a connection back to the top level search of the database.
(Currently I'm using a modified custom processor, LookupAttributeFromSQL, that Brett Ryan
did in January 18th, before he worked on a SQLLookupService.)

The problem is that I could have 2 records coming in with the same pieces of information,
and because it's flow based, the check for the 4 table join will come up empty on the second
record before the first record is done creating the 4 table records. I've investigated the
Wait/Notify pattern, but the odd part for me is that you need to have a separate "initialization"
of the Wait/Notify release signal indicator (
and that seems "hack-ish" to me.

With all of that said, I was curious if there was a way to have a back pressure value of 1
into the Process Group, so that if there is a flow file anywhere in the Process Group, the
flow file is unable to enter that Process Group? That way, the creation of the 4 records could
be inside a process group, and no other flowfile can enter until that first flowfile has exited.

Thanks for any insight,
View raw message