nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Wicks (pwicks)" <pwi...@micron.com>
Subject RE: [EXT] Sliding windows
Date Tue, 04 Jun 2019 13:50:42 GMT
Craig,

If you have a timestamp set as an attribute on the processor, then this is kind of possible.

Have a regular MergeContent processor, with "Maximum Group Size" set to 1 mb, set "Max Bin
Age" to 3 min; you may need to tweak settings to get the right cadence, but these are generally
the settings you need to touch. Use the "Merged" relationship for whatever you need. To create
the Window, pass the "Original" relationship to a RouteOnAttribute processor.

In the RouteOnAttribute use NiFi Expression Language to calculate how old the FlowFile is
(using the timestamp attribute I mentioned). If the FlowFile is older than x, drop it, else
send it back to the MergeContent processor.

Using this process, it should be easy to get a 5 min rolling window (drop any FlowFile older
than 5 min in RouteOnAttribute).

I don't know that this perfectly answers what you asked, but does it give you a good direction
to investigate?

Thanks,
  Peter

-----Original Message-----
From: Craig Knell <craig.knell@gmail.com> 
Sent: Tuesday, June 4, 2019 1:32 AM
To: dev@nifi.apache.org
Subject: [EXT] Sliding windows

Hi Folks

We have a stream of data that I need to window to 5 minutes and the window is to slide every
3 minutes. Each minute is 1 mb, I therefore have to deliver 5mb per 3 minutes.  

What is the best way of achieving this in nifi?

Best regards

Craig

Mime
View raw message