nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John McGinn <amruginn-n...@yahoo.com>
Subject Back Pressure on Process Group
Date Thu, 11 Oct 2018 18:17:21 GMT
I've been going through mailing list archives, and looking at blog posts around Wait/Notify,
and these don't seem to be the solution for my use case.

My basic use case is as follows. I have 4 DB tables, 3 of which are id/name pairs (office
name, city, state), and the 4th table joins the 3 ids together to a new id which is used elsewhere
in the database.

Using NiFi to injest data from a different database system, we have to verify if that office
is active, and if it isn't active or non-existent, create a new record, as well as any of
the other 3 tables necessary.

The first step, then, is to join the 4 tables together to search for the name fields, and
if the join comes back with a row, use that top level id as an attribute. No problem, works
fine. (FetchDatabaseTable -> AvroToJson -> EvaluateJsonPath, etc.) If the join comes
back empty, I need to insert rows for the 3 pieces and then join them together. Ideally, this
would be a flow of 3 PutSqls, then a connection back to the top level search of the database.
(Currently I'm using a modified custom processor, LookupAttributeFromSQL, that Brett Ryan
did in January 18th, before he worked on a SQLLookupService.)

The problem is that I could have 2 records coming in with the same pieces of information,
and because it's flow based, the check for the 4 table join will come up empty on the second
record before the first record is done creating the 4 table records. I've investigated the
Wait/Notify pattern, but the odd part for me is that you need to have a separate "initialization"
of the Wait/Notify release signal indicator (https://gist.github.com/ijokarumawak/9e1a4855934f2bb9661f88ca625bd244)
and that seems "hack-ish" to me.

With all of that said, I was curious if there was a way to have a back pressure value of 1
into the Process Group, so that if there is a flow file anywhere in the Process Group, the
flow file is unable to enter that Process Group? That way, the creation of the 4 records could
be inside a process group, and no other flowfile can enter until that first flowfile has exited.


Thanks for any insight,
John

Mime
View raw message