nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy LoPresto <alopre...@apache.org>
Subject Re: Having a processor wait for all inputs
Date Thu, 16 Mar 2017 02:01:31 GMT
Hi Mike,

Depending on the specifics of the inputs (source, format, correlating attributes, etc.) it
sounds like you have a few good options to achieve this.

1. MergeContent [1] — This processor is very old (read: stable & widely-used, not legacy)
and is used to merge content from multiple incoming flowfiles into batches according to a
variety of strategies. There are many threads on these lists and articles/Stack Overflow/HCC
posts about it, as it is very versatile.
2. Wait/Notify [2] — New in NiFi 1.2.0 (not yet released but available currently on the
master branch), Joe Gresock and Koji Kawamura have contributed some very useful processors
which allow you to wait for an arbitrary number of “signals” to invoke further processors.
See the PRs [3][4][5] and excellent explanation [6].
3. ExecuteScript — If neither of those options work for you, although I suspect they will,
you can write custom code in any of 6 languages (Ruby, Python, Lua, Javascript, Groovy, or
Java) and run it with the ExecuteScript or InvokeScriptedProcessor to perform arbitrary operations
without having to build a complete NAR and custom processors. Of course, you can still do
that if you desire.

Hope that helps. Please let us know if you have any other questions.

[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MergeContent/index.html
<https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MergeContent/index.html>
[2] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/Wait.java
<https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/Wait.java>
[3] https://github.com/apache/nifi/pull/1329 <https://github.com/apache/nifi/pull/1329>
[4] https://github.com/apache/nifi/pull/1420 <https://github.com/apache/nifi/pull/1420>
[5] https://github.com/apache/nifi/pull/1490 <https://github.com/apache/nifi/pull/1490>
[6] https://gist.github.com/ijokarumawak/85a3d77297ea94614e9f3f2a9dabca67 <https://gist.github.com/ijokarumawak/85a3d77297ea94614e9f3f2a9dabca67>

Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Mar 15, 2017, at 6:38 PM, Michael John LEWIS <michaellewis@dbs.com> wrote:
> 
> Hi,
> 
> I’ve just started taking a look at Nifi as it seems perfect for a project I’m working
on right now.
> 
> The one thing I’m trying to solve is this – I have a process (say a batch job or
similar) that I want to invoke,
> only once all the inputs have arrived ( a simple dependency graph ).
> 
> i.e. I have 3 files and I don’t want to run a process after each file arrival, but
once all 3 files have arrived.
> 
> Is this easily achievable in Nifi ? Or would I have to think about custom processor?
> 
> Thanks in advance,
> 
> Mike
> 
> 
> CONFIDENTIAL NOTE:
> 
> The information contained in this email is intended only for the use of the individual
or entity named above and may contain information that is privileged, confidential and exempt
from disclosure under applicable law. If the reader of this message is not the intended recipient,
you are hereby notified that any dissemination, distribution or copying of this communication
is strictly prohibited. If you have received this message in error, please immediately notify
the sender and delete the mail. Thank you.


Mime
View raw message