nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: MiNiFi event-based scheduling?
Date Fri, 06 Jan 2017 15:59:01 GMT
Andy,

Of note - with NiFi, the way that the Timer-Based Scheduling agent works, if you set a schedule
of "0 secs"
NiFi will determine whether or not there is any work that can be done by the processor. If
the processor has
no work to do, it is considered "bored" and isn't scheduled again for 10 milliseconds (this
is configurable in
nifi.properties, using the "nifi.bored.yield.duration" property, of which the default is 10
milliseconds). This is
designed to avoid hammering the CPU.

Off the top of my head, I am not sure if MiNiFi supports this feature or not, though. Any
idea, Aldrin?

Thanks
-Mark


> On Jan 6, 2017, at 10:51 AM, Aldrin Piri <aldrinpiri@gmail.com> wrote:
> 
> Great!  Thanks for taking the time to do so, much appreciated.
> 
> On Fri, Jan 6, 2017 at 10:48 AM, Andrew Christianson <
> andrew.christianson@nextcentury.com> wrote:
> 
>> Got it. Added this as MINIFI-182 with a link to the wiki for design ideas.
>> 
>> ________________________________________
>> From: Aldrin Piri <aldrinpiri@gmail.com>
>> Sent: Friday, January 6, 2017 10:03:06 AM
>> To: dev
>> Subject: Re: MiNiFi event-based scheduling?
>> 
>> Hey Andy,
>> 
>> Your inspection would prove correct and it does not appear we have a JIRA
>> as of yet either.  If you would be interested in creating one, please do,
>> otherwise I shall see to it that it gets added.
>> 
>> We are still working to get some of the more foundational framework
>> components established but certainly can see how this would be helpful.
>> Definitely should track this so that it isn't lost in the shuffle and the
>> framework can be refactored to support it.
>> 
>> Thanks!
>> --aldrin
>> 
>> On Fri, Jan 6, 2017 at 9:56 AM, Andrew Christianson <
>> andrew.christianson@nextcentury.com> wrote:
>> 
>>> We noticed that MiNiFi currently only supports timer-based scheduling.
>>> Looking at the code, one could set the timer interval to 0, but this
>> would
>>> result in a hot loop burning 100% of a core. As such, there seems to be
>> no
>>> way presently to have flow file records processed as soon as incoming
>>> records populate the queue. Further, it looks like the FlowController
>> code
>>> is hard-coded against the timer-based scheduling agent, and unfortunately
>>> there is no obvious way to plug-in an alternative scheduler.
>>> 
>>> Are there any near/mid-term plans to implement event-based scheduling?
>>> 
>>> -Andy
>>> 
>> 


Mime
View raw message