nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Zhurakousky <ozhurakou...@hortonworks.com>
Subject Re: Delay Processor
Date Tue, 15 Nov 2016 14:05:43 GMT
I am +1 on this as I’ve seen many cases in the field where it is applicable and as you mention
exponential back off is one of the common one. That said, I am wondering if that has to be
a processor at all? Actually let me answer my own question. There are definitely cases where
it has to be a processor. Those are true delay with intention requirements (i.e., Compute
Andrew’s greetings -> Delay until his b-day -> Send Greetings). 
The exponential back off is a bit different since it almost aligns with circuit breaker and
re-tries. Currently we simply retry by resubmitting the flow file with fixed yield. However
one may argue that if something failed the first time, it is very likely that it is going
to fail again and again. It may also succeed, but one may argue that it has a higher chance
of succeeding after certain delay which increases on subsequent failures until we may choose
to consider it a failed cause and stop resubmitting it.

So in summary I see the two-part requirement;  A processor and enhancement to the core-framework’s
retry logic.

Thoughts?
Cheers
Oleg


> On Nov 15, 2016, at 8:50 AM, Andrew Grande <aperepel@gmail.com> wrote:
> 
> Hi,
> 
> I'd lime to check where discussions are on this, ir propose the new component otherwise.
> 
> Use case: make delay strategies explicit, easier to use. E.g. think of a failure retry
loop.
> 
> Currently, ControlRate is somewhat related, but can be improved. E.g. introduce delay
strategies a la prioritizers on the connection?
> 
> Thinking out loud, something like a exponential backoff strategy could be kept stateless
by adding a number of housekeeping attributes to the FF, which eliminates the need for any
state in the processor itself.
> 
> I'll stop here to see if any ideas were captured prior abd what community thinks of it.
> 
> Andrew

Mime
View raw message