nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: NiFI Custom Plugin - Yield Duration Ignored
Date Tue, 17 Nov 2015 16:19:22 GMT
Michael,

Hello. First, thanks for your interest in NiFi! I'm very happy to see that you're already
starting to dig in pretty deep here!

I admit that the wording here can be a little confusing. Let me try to explain what's going
on here and how yielding
works. When you are developing a Processor in NiFi, especially if reaching out to some external
resource,
a common occurrence is to realize that there is no work that can really be done at this point
in time (for
example, because the remote resource is not accessible). In that case, the processor can call
ProcessContext.yield()
and as a result, the framework will stop scheduling that processor to run for the configured
amount of time, so that
we aren't wasting resources when we know there is nothing that we can accomplish. This is
where the configured
Yield Duration comes in.

There is, however, a concept of "administratively yielding" a processor. The idea here is
that the Processor throws
some unexpected exception. Since the Exception was not of type ProcessException, the framework
considers this
a bug and will perform an administrative yield to avoid continually triggering a processor
that may be in a bad state.

Perhaps using the term "administratively yielding" was a bad idea, as it can clearly lead
to confusion.
If you feel that something is not behaving properly, though, or you believe that the user
experience can be improved
in some way, please do let us know, as we strive to continually improve the user experience.

Thanks
-Mark


> On Nov 17, 2015, at 10:34 AM, Michael De Courci <mdecourci@gmail.com> wrote:
> 
> FYI
> 
>> Begin forwarded message:
>> 
>> From: Michael De Courci <mdecourci@googlemail.com>
>> Date: 16 November 2015 at 15:41:27 GMT
>> To: users@nifi.apache.org, Michael.DeCourci@barclayscapital.com, Kefah Issa <kefah.issa@barclays.com>
>> Subject: NiFI Custom Plugin - Yield Duration Ignored
>> 
>> I wrote a NAR custom plugin to verify NoFi Exception handling. - version; nifi-0.3.0
>> 
>> The plugin was written to always through a RunTimeException for any exception including
ProcessException.
>> 
>> I created a simple flow; GetFile->MyPlugin->PutFile 
>> 
>> and set the Yield duration to 60 seconds.
>> 
>> The log shows that the yield duration is always 1 sec
>> 
>> I debugged the NiFi framework code and see that the yield duration always defaults
to 1 sec;
>> See;
>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent
>> 
>> Setting the duration updates the class 
>> org.apache.nifi.controller.StandardProcessorNode
>> but this does not communicate the change to TimerDrivenSchedulingAgent
>> 
>> Evidence from log;
>> 
>> 80eb-552a-471c-8ffd-f2509d71fd2e] failed to process session due to java.lang.RuntimeException:
java.lang.RuntimeException
>> 
>> 2015-11-15 16:17:58,417 WARN [Timer-Driven Process Thread-7] c.b.n.p.SampleErrorHandlerProcessor
SampleErrorHandlerProcessor[id=8e8f80eb-552a-471c-8ffd-f2509d71fd2e] Processor Administratively
Yielded for 1 sec due to processing failure
>> 
>> 2015-11-15 16:17:58,417 WARN [Timer-Driven Process Thread-7] o.a.n.c.t.ContinuallyRunProcessorTask
Administratively Yielding SampleErrorHandlerProcessor[id=8e8f80eb-552a-471c-8ffd-f2509d71fd2e]
due to uncaught Exception: java.lang.RuntimeException
>> 
>> 2015-11-15 16:17:58,419 WARN [Timer-Driven Process Thread-7] o.a.n.c.t.ContinuallyRunProcessorTask
>> 
>> java.lang.RuntimeException: null
>> 
>>        at com.barclays.nifi.processor.SampleErrorHandlerProcessor.onTrigger(SampleErrorHandlerProcessor.scala:103)
~[na:na]
>> 
>> 
>> 
>> With Kind Regards
>> Michael de Courci
> 


Mime
View raw message