nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael De Courci <mdecou...@gmail.com>
Subject Re: NiFI Custom Plugin - Yield Duration Ignored
Date Tue, 17 Nov 2015 16:51:06 GMT
Is it possible to interrupt/suspend the whole flow (upstream as well as downstream) when a
plugin encounters an error?

Michael

> On 17 Nov 2015, at 16:46, Oleg Zhurakousky <ozhurakousky@hortonworks.com> wrote:
> 
> Perhaps we may need to start explaining ‘administrative yield” as Circuit Breaker
because that ’s what it is from known pattern perspective
> 
> Question to DEV; Should we consider going as far as deprecating the name and incorporating
CB in it somehow? That would definitely help to avoid the confusion
> Cheers
> Oleg
> 
>> On Nov 17, 2015, at 11:19 AM, Mark Payne <markap14@hotmail.com> wrote:
>> 
>> Michael,
>> 
>> Hello. First, thanks for your interest in NiFi! I'm very happy to see that you're
already starting to dig in pretty deep here!
>> 
>> I admit that the wording here can be a little confusing. Let me try to explain what's
going on here and how yielding
>> works. When you are developing a Processor in NiFi, especially if reaching out to
some external resource,
>> a common occurrence is to realize that there is no work that can really be done at
this point in time (for
>> example, because the remote resource is not accessible). In that case, the processor
can call ProcessContext.yield()
>> and as a result, the framework will stop scheduling that processor to run for the
configured amount of time, so that
>> we aren't wasting resources when we know there is nothing that we can accomplish.
This is where the configured
>> Yield Duration comes in.
>> 
>> There is, however, a concept of "administratively yielding" a processor. The idea
here is that the Processor throws
>> some unexpected exception. Since the Exception was not of type ProcessException,
the framework considers this
>> a bug and will perform an administrative yield to avoid continually triggering a
processor that may be in a bad state.
>> 
>> Perhaps using the term "administratively yielding" was a bad idea, as it can clearly
lead to confusion.
>> If you feel that something is not behaving properly, though, or you believe that
the user experience can be improved
>> in some way, please do let us know, as we strive to continually improve the user
experience.
>> 
>> Thanks
>> -Mark
>> 
>> 
>>> On Nov 17, 2015, at 10:34 AM, Michael De Courci <mdecourci@gmail.com> wrote:
>>> 
>>> FYI
>>> 
>>>> Begin forwarded message:
>>>> 
>>>> From: Michael De Courci <mdecourci@googlemail.com>
>>>> Date: 16 November 2015 at 15:41:27 GMT
>>>> To: users@nifi.apache.org, Michael.DeCourci@barclayscapital.com, Kefah Issa
<kefah.issa@barclays.com>
>>>> Subject: NiFI Custom Plugin - Yield Duration Ignored
>>>> 
>>>> I wrote a NAR custom plugin to verify NoFi Exception handling. - version;
nifi-0.3.0
>>>> 
>>>> The plugin was written to always through a RunTimeException for any exception
including ProcessException.
>>>> 
>>>> I created a simple flow; GetFile->MyPlugin->PutFile 
>>>> 
>>>> and set the Yield duration to 60 seconds.
>>>> 
>>>> The log shows that the yield duration is always 1 sec
>>>> 
>>>> I debugged the NiFi framework code and see that the yield duration always
defaults to 1 sec;
>>>> See;
>>>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent
>>>> 
>>>> Setting the duration updates the class 
>>>> org.apache.nifi.controller.StandardProcessorNode
>>>> but this does not communicate the change to TimerDrivenSchedulingAgent
>>>> 
>>>> Evidence from log;
>>>> 
>>>> 80eb-552a-471c-8ffd-f2509d71fd2e] failed to process session due to java.lang.RuntimeException:
java.lang.RuntimeException
>>>> 
>>>> 2015-11-15 16:17:58,417 WARN [Timer-Driven Process Thread-7] c.b.n.p.SampleErrorHandlerProcessor
SampleErrorHandlerProcessor[id=8e8f80eb-552a-471c-8ffd-f2509d71fd2e] Processor Administratively
Yielded for 1 sec due to processing failure
>>>> 
>>>> 2015-11-15 16:17:58,417 WARN [Timer-Driven Process Thread-7] o.a.n.c.t.ContinuallyRunProcessorTask
Administratively Yielding SampleErrorHandlerProcessor[id=8e8f80eb-552a-471c-8ffd-f2509d71fd2e]
due to uncaught Exception: java.lang.RuntimeException
>>>> 
>>>> 2015-11-15 16:17:58,419 WARN [Timer-Driven Process Thread-7] o.a.n.c.t.ContinuallyRunProcessorTask
>>>> 
>>>> java.lang.RuntimeException: null
>>>> 
>>>>      at com.barclays.nifi.processor.SampleErrorHandlerProcessor.onTrigger(SampleErrorHandlerProcessor.scala:103)
~[na:na]
>>>> 
>>>> 
>>>> 
>>>> With Kind Regards
>>>> Michael de Courci
>>> 
>> 
>> 
> 


Mime
View raw message