nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: Delay Processor
Date Tue, 15 Nov 2016 14:06:32 GMT
The concept of flowfile penalization exists to support many of the
requirements for delay.  This is something which can be set, by a
component (processor), on a flow file for some given period of time
and which will be honored on the connection it is placed on such that
things without this penalization will be higher priority.  It was
built for the typical failure loops.  This is appropriate when it is
something about that 'flowfile' and the thing the processor it trying
to interact with that is the problem and therefore simply marking that
flowfile as problematic for a bit is likely to allow whatever the
condition is to go away without just making the problem worse.

There is also the concept of process yielding.  A component
(processor) can signal that it should yield which means it will not be
triggered again at its normally scheduled time but rather will yield
until its yield period is up.  This makes sense when there is
something about the state of that processor and its configuration
relative to whatever it is trying to do/interacting with which is
expected to be able to go away or be resolved on its own.

Mark Payne and I have talked in the past about the notion of the
framework automatically tracking flowfiles that appear to be looping
and doing exponential backoff on them.  This would be done just as you
suggested which is via flow file attributes.

Is your case/concern more like the 'something is wrong with this flow
file vs environment' or more like 'something is wrong with this
process vs environment'?



On Tue, Nov 15, 2016 at 8:50 AM, Andrew Grande <aperepel@gmail.com> wrote:
> Hi,
>
> I'd lime to check where discussions are on this, ir propose the new
> component otherwise.
>
> Use case: make delay strategies explicit, easier to use. E.g. think of a
> failure retry loop.
>
> Currently, ControlRate is somewhat related, but can be improved. E.g.
> introduce delay strategies a la prioritizers on the connection?
>
> Thinking out loud, something like a exponential backoff strategy could be
> kept stateless by adding a number of housekeeping attributes to the FF,
> which eliminates the need for any state in the processor itself.
>
> I'll stop here to see if any ideas were captured prior abd what community
> thinks of it.
>
> Andrew

Mime
View raw message