nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aldrin Piri <aldrinp...@gmail.com>
Subject Re: NiFI Custom Plugin - Yield Duration Ignored
Date Tue, 17 Nov 2015 16:54:52 GMT
I appreciate the use of patterns, but given that non-devs will often be
interepeting these types of issues, I think the name is a miss in
recognizability.  Regardless, we can and should do better about clarifying
the language in the logs and docs.

On Tue, Nov 17, 2015 at 11:46 AM, Oleg Zhurakousky <
ozhurakousky@hortonworks.com> wrote:

> Perhaps we may need to start explaining ‘administrative yield” as Circuit
> Breaker because that ’s what it is from known pattern perspective
>
> Question to DEV; Should we consider going as far as deprecating the name
> and incorporating CB in it somehow? That would definitely help to avoid the
> confusion
> Cheers
> Oleg
>
> > On Nov 17, 2015, at 11:19 AM, Mark Payne <markap14@hotmail.com> wrote:
> >
> > Michael,
> >
> > Hello. First, thanks for your interest in NiFi! I'm very happy to see
> that you're already starting to dig in pretty deep here!
> >
> > I admit that the wording here can be a little confusing. Let me try to
> explain what's going on here and how yielding
> > works. When you are developing a Processor in NiFi, especially if
> reaching out to some external resource,
> > a common occurrence is to realize that there is no work that can really
> be done at this point in time (for
> > example, because the remote resource is not accessible). In that case,
> the processor can call ProcessContext.yield()
> > and as a result, the framework will stop scheduling that processor to
> run for the configured amount of time, so that
> > we aren't wasting resources when we know there is nothing that we can
> accomplish. This is where the configured
> > Yield Duration comes in.
> >
> > There is, however, a concept of "administratively yielding" a processor.
> The idea here is that the Processor throws
> > some unexpected exception. Since the Exception was not of type
> ProcessException, the framework considers this
> > a bug and will perform an administrative yield to avoid continually
> triggering a processor that may be in a bad state.
> >
> > Perhaps using the term "administratively yielding" was a bad idea, as it
> can clearly lead to confusion.
> > If you feel that something is not behaving properly, though, or you
> believe that the user experience can be improved
> > in some way, please do let us know, as we strive to continually improve
> the user experience.
> >
> > Thanks
> > -Mark
> >
> >
> >> On Nov 17, 2015, at 10:34 AM, Michael De Courci <mdecourci@gmail.com>
> wrote:
> >>
> >> FYI
> >>
> >>> Begin forwarded message:
> >>>
> >>> From: Michael De Courci <mdecourci@googlemail.com>
> >>> Date: 16 November 2015 at 15:41:27 GMT
> >>> To: users@nifi.apache.org, Michael.DeCourci@barclayscapital.com,
> Kefah Issa <kefah.issa@barclays.com>
> >>> Subject: NiFI Custom Plugin - Yield Duration Ignored
> >>>
> >>> I wrote a NAR custom plugin to verify NoFi Exception handling. -
> version; nifi-0.3.0
> >>>
> >>> The plugin was written to always through a RunTimeException for any
> exception including ProcessException.
> >>>
> >>> I created a simple flow; GetFile->MyPlugin->PutFile
> >>>
> >>> and set the Yield duration to 60 seconds.
> >>>
> >>> The log shows that the yield duration is always 1 sec
> >>>
> >>> I debugged the NiFi framework code and see that the yield duration
> always defaults to 1 sec;
> >>> See;
> >>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent
> >>>
> >>> Setting the duration updates the class
> >>> org.apache.nifi.controller.StandardProcessorNode
> >>> but this does not communicate the change to TimerDrivenSchedulingAgent
> >>>
> >>> Evidence from log;
> >>>
> >>> 80eb-552a-471c-8ffd-f2509d71fd2e] failed to process session due to
> java.lang.RuntimeException: java.lang.RuntimeException
> >>>
> >>> 2015-11-15 16:17:58,417 WARN [Timer-Driven Process Thread-7]
> c.b.n.p.SampleErrorHandlerProcessor
> SampleErrorHandlerProcessor[id=8e8f80eb-552a-471c-8ffd-f2509d71fd2e]
> Processor Administratively Yielded for 1 sec due to processing failure
> >>>
> >>> 2015-11-15 16:17:58,417 WARN [Timer-Driven Process Thread-7]
> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
> SampleErrorHandlerProcessor[id=8e8f80eb-552a-471c-8ffd-f2509d71fd2e] due to
> uncaught Exception: java.lang.RuntimeException
> >>>
> >>> 2015-11-15 16:17:58,419 WARN [Timer-Driven Process Thread-7]
> o.a.n.c.t.ContinuallyRunProcessorTask
> >>>
> >>> java.lang.RuntimeException: null
> >>>
> >>>       at
> com.barclays.nifi.processor.SampleErrorHandlerProcessor.onTrigger(SampleErrorHandlerProcessor.scala:103)
> ~[na:na]
> >>>
> >>>
> >>>
> >>> With Kind Regards
> >>> Michael de Courci
> >>
> >
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message