nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: Communication between flows
Date Wed, 05 Jul 2017 17:01:15 GMT
Steve,

In 1.2.0 there were some new processors added called Wait/Notify...

With those you could send your original JSON (before splitting) to a
Wait processor and tell it to wait until the signal count is equal to
the number of splits, then you could put a Notify processor right
after PutMongo connected to the success relationship. For example, if
100 JSON documents get split out, the Wait processor is waiting for
100 signals or until it times out, and signals are only sent after
successful insertion to Mongo. You can checkout this blog for an
example [1].

In 1.1.2, you might be able to put a MergeContent processor configured
to run in defragment mode connected to the success relationship of
PutMongo. Defragment mode is used to undo the splitting that was done
by an unpstream processors, so it will only defragment and merged back
together if all the fragments made it through.

-Bryan

[1] https://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/


On Wed, Jul 5, 2017 at 12:46 PM, Byers, Steven K (Steve) CTR USARMY
MEDCOM JMLFDC (US) <steven.k.byers.ctr@mail.mil> wrote:
> Is there a mechanism or technique for communicating the results of a flow file to its
"sister" flow files?
>
> Here is a high-level description of what I am doing:
>
> Input to my flow is a JSON array of documents that get split (SplitJson) into individual
documents and each document becomes a distinct flow file.  Each document (flow file) gets
validated against a JSON schema (ValidateJson) then gets updated into a Mongo collection (PutMongoUpdate).
 At the end of all this, I want to do some post processing but only if all documents processed
successfully.
>
> When a failure occurs (either in the validation or the Mongo update) is there a way to
communicate that to the success branch of the flow process so a decision can be made about
whether to proceed to post processing or not.
>
> I am using NiFi 1.1.2
>
> Thank you for any guidance you can offer,
>
> Steve
>
>
>

Mime
View raw message