nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elli Schwarz <eliezer_schw...@yahoo.com>
Subject Queue wasn't emptying
Date Fri, 06 Nov 2015 16:25:49 GMT
I had a queue that built up overnight with several thousand flowfiles. The queue was pointing
to a RouteOnAttribute processor, which was running. For some reason, the RouteOnAttribute
processor wasn't emptying the queue so the queue just built up. Stopping and starting the
processor didn't help. However, simply restarting Nifi got the flow moving again. I can't
figure out why the queue was stuck.

What could cause a queue to build up like that? How would restarting Nifi get it going again?
There weren't any errors in the log.  I did have backpressure on the relationship to the
processor after the RouteOnAttribute set to 1, but that queue was empty. There was no backpressure
on the queue heading in to the RouteOnAttribute processor. The RouteOnAttribute processor
has a Penalty Duration and a Yield of 1 sec. The relationships all used the PriorityAttributeAnalyzer
as the prioritizer. 

The interesting thing is I had the exact same problem on two different nifis (one was version
0.2.0 and one 0.3.0). The same type of flow files were going from one to the other using site-to-site
(I was doing this for load balancing). On the other Nifi, instead of restarting, I added another
RouteOnAttribute processor and rerouted the stuck queue to the new one instead, and that also
got things moving again. (I have a backpressure since the processor after the RouteOnAttribute
is a ControlRate processor, and I only want to allow 1 flowfile through per second so as not
to overwhelm the system downstream. The backpressure is to assist with load balancing, so
if one pathway fills up, I have the other pathway which uses the site-to-site to route the
flow to the other nifi).

Is there some combination of backpressure, penalty or yield along with prioritization that
could cause a kind of deadlock-like situation? Any ideas as to how I can prevent this from
occurring?
Thanks!-Elli


Mime
View raw message