nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Zhurakousky <ozhurakou...@hortonworks.com>
Subject Re: NiFI Custom Plugin - Yield Duration Ignored
Date Tue, 17 Nov 2015 17:02:19 GMT
Aldrin
If it was just a software pattern and given the target audience of NiFi I’d agree with you
whole heartedly.
But CB is something everyone understands since every one has a few in their household ;)

Cheers
Oleg


> On Nov 17, 2015, at 11:54 AM, Aldrin Piri <aldrinpiri@gmail.com> wrote:
> 
> I appreciate the use of patterns, but given that non-devs will often be
> interepeting these types of issues, I think the name is a miss in
> recognizability.  Regardless, we can and should do better about clarifying
> the language in the logs and docs.
> 
> On Tue, Nov 17, 2015 at 11:46 AM, Oleg Zhurakousky <
> ozhurakousky@hortonworks.com> wrote:
> 
>> Perhaps we may need to start explaining ‘administrative yield” as Circuit
>> Breaker because that ’s what it is from known pattern perspective
>> 
>> Question to DEV; Should we consider going as far as deprecating the name
>> and incorporating CB in it somehow? That would definitely help to avoid the
>> confusion
>> Cheers
>> Oleg
>> 
>>> On Nov 17, 2015, at 11:19 AM, Mark Payne <markap14@hotmail.com> wrote:
>>> 
>>> Michael,
>>> 
>>> Hello. First, thanks for your interest in NiFi! I'm very happy to see
>> that you're already starting to dig in pretty deep here!
>>> 
>>> I admit that the wording here can be a little confusing. Let me try to
>> explain what's going on here and how yielding
>>> works. When you are developing a Processor in NiFi, especially if
>> reaching out to some external resource,
>>> a common occurrence is to realize that there is no work that can really
>> be done at this point in time (for
>>> example, because the remote resource is not accessible). In that case,
>> the processor can call ProcessContext.yield()
>>> and as a result, the framework will stop scheduling that processor to
>> run for the configured amount of time, so that
>>> we aren't wasting resources when we know there is nothing that we can
>> accomplish. This is where the configured
>>> Yield Duration comes in.
>>> 
>>> There is, however, a concept of "administratively yielding" a processor.
>> The idea here is that the Processor throws
>>> some unexpected exception. Since the Exception was not of type
>> ProcessException, the framework considers this
>>> a bug and will perform an administrative yield to avoid continually
>> triggering a processor that may be in a bad state.
>>> 
>>> Perhaps using the term "administratively yielding" was a bad idea, as it
>> can clearly lead to confusion.
>>> If you feel that something is not behaving properly, though, or you
>> believe that the user experience can be improved
>>> in some way, please do let us know, as we strive to continually improve
>> the user experience.
>>> 
>>> Thanks
>>> -Mark
>>> 
>>> 
>>>> On Nov 17, 2015, at 10:34 AM, Michael De Courci <mdecourci@gmail.com>
>> wrote:
>>>> 
>>>> FYI
>>>> 
>>>>> Begin forwarded message:
>>>>> 
>>>>> From: Michael De Courci <mdecourci@googlemail.com>
>>>>> Date: 16 November 2015 at 15:41:27 GMT
>>>>> To: users@nifi.apache.org, Michael.DeCourci@barclayscapital.com,
>> Kefah Issa <kefah.issa@barclays.com>
>>>>> Subject: NiFI Custom Plugin - Yield Duration Ignored
>>>>> 
>>>>> I wrote a NAR custom plugin to verify NoFi Exception handling. -
>> version; nifi-0.3.0
>>>>> 
>>>>> The plugin was written to always through a RunTimeException for any
>> exception including ProcessException.
>>>>> 
>>>>> I created a simple flow; GetFile->MyPlugin->PutFile
>>>>> 
>>>>> and set the Yield duration to 60 seconds.
>>>>> 
>>>>> The log shows that the yield duration is always 1 sec
>>>>> 
>>>>> I debugged the NiFi framework code and see that the yield duration
>> always defaults to 1 sec;
>>>>> See;
>>>>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent
>>>>> 
>>>>> Setting the duration updates the class
>>>>> org.apache.nifi.controller.StandardProcessorNode
>>>>> but this does not communicate the change to TimerDrivenSchedulingAgent
>>>>> 
>>>>> Evidence from log;
>>>>> 
>>>>> 80eb-552a-471c-8ffd-f2509d71fd2e] failed to process session due to
>> java.lang.RuntimeException: java.lang.RuntimeException
>>>>> 
>>>>> 2015-11-15 16:17:58,417 WARN [Timer-Driven Process Thread-7]
>> c.b.n.p.SampleErrorHandlerProcessor
>> SampleErrorHandlerProcessor[id=8e8f80eb-552a-471c-8ffd-f2509d71fd2e]
>> Processor Administratively Yielded for 1 sec due to processing failure
>>>>> 
>>>>> 2015-11-15 16:17:58,417 WARN [Timer-Driven Process Thread-7]
>> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
>> SampleErrorHandlerProcessor[id=8e8f80eb-552a-471c-8ffd-f2509d71fd2e] due to
>> uncaught Exception: java.lang.RuntimeException
>>>>> 
>>>>> 2015-11-15 16:17:58,419 WARN [Timer-Driven Process Thread-7]
>> o.a.n.c.t.ContinuallyRunProcessorTask
>>>>> 
>>>>> java.lang.RuntimeException: null
>>>>> 
>>>>>      at
>> com.barclays.nifi.processor.SampleErrorHandlerProcessor.onTrigger(SampleErrorHandlerProcessor.scala:103)
>> ~[na:na]
>>>>> 
>>>>> 
>>>>> 
>>>>> With Kind Regards
>>>>> Michael de Courci
>>>> 
>>> 
>>> 
>> 
>> 

Mime
View raw message