Hi Mike,

Depending on the specifics of the inputs (source, format, correlating attributes, etc.) it sounds like you have a few good options to achieve this. 

1. MergeContent [1] — This processor is very old (read: stable & widely-used, not legacy) and is used to merge content from multiple incoming flowfiles into batches according to a variety of strategies. There are many threads on these lists and articles/Stack Overflow/HCC posts about it, as it is very versatile. 
2. Wait/Notify [2] — New in NiFi 1.2.0 (not yet released but available currently on the master branch), Joe Gresock and Koji Kawamura have contributed some very useful processors which allow you to wait for an arbitrary number of “signals” to invoke further processors. See the PRs [3][4][5] and excellent explanation [6]. 
3. ExecuteScript — If neither of those options work for you, although I suspect they will, you can write custom code in any of 6 languages (Ruby, Python, Lua, Javascript, Groovy, or Java) and run it with the ExecuteScript or InvokeScriptedProcessor to perform arbitrary operations without having to build a complete NAR and custom processors. Of course, you can still do that if you desire. 

Hope that helps. Please let us know if you have any other questions. 

[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MergeContent/index.html
[2] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/Wait.java
[3] https://github.com/apache/nifi/pull/1329
[4] https://github.com/apache/nifi/pull/1420
[5] https://github.com/apache/nifi/pull/1490
[6] https://gist.github.com/ijokarumawak/85a3d77297ea94614e9f3f2a9dabca67

Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Mar 15, 2017, at 6:38 PM, Michael John LEWIS <michaellewis@dbs.com> wrote:


I’ve just started taking a look at Nifi as it seems perfect for a project I’m working on right now.

The one thing I’m trying to solve is this – I have a process (say a batch job or similar) that I want to invoke,
only once all the inputs have arrived ( a simple dependency graph ).

i.e. I have 3 files and I don’t want to run a process after each file arrival, but once all 3 files have arrived.

Is this easily achievable in Nifi ? Or would I have to think about custom processor?

Thanks in advance,



The information contained in this email is intended only for the use of the individual or entity named above and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete the mail. Thank you.