nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Josef.Zahn...@swisscom.com>
Subject Re: NiFi 1.8 and stuck flowfile in Load Balanced enabled queue
Date Mon, 19 Nov 2018 13:23:39 GMT
@Mark: I’m now able to query the REST-API. However, the call nifi-api/processors/<processor-id>/diagnostics
does reply on all nodes “An unexpected error has occurred. Please check the logs for additional
details.” If I remove the /diagnostics I’m getting a useful reply in JSON format. I tried
on several nifi nodes, even on a not secure one (http://unsec.company.net:8080/nifi-api/processors/eeba4842-0166-1000-9f56-e59ad010d040/diagnostics
) , all show the same behavior with the error.

Example:
curl 'https://abc.company.net:8443/nifi-api/processors/cea533b0-edb9-1d11-955f-7380c973ac62/diagnostics'
-H 'Authorization: Bearer $TOKEN' --compressed –insecure
An unexpected error has occurred. Please check the logs for additional details.

The loadbalancing problem still exists on our cluster…

Cheers Josef

From: "Zahner Josef, GSB-LR-TRW-LI" <Josef.Zahner1@swisscom.com>
Date: Friday, 16 November 2018 at 22:32
To: "users@nifi.apache.org" <users@nifi.apache.org>
Subject: Re: NiFi 1.8 and stuck flowfile in Load Balanced enabled queue

Looks pretty much the same in our case, just that we have a 8 node cluster (all nodes connected).


  *   Flowfile expiration: 0
  *   Back Pressure Object Threshold: 1000000
  *   Size Threshold: 1 GB
  *   Load Balance Strategy: Round Robin
  *   Load Balance Compression: Do not compress


We have on both sides a processor. Just tried to access the nifi-api for the destination,
but we have a secured (ldap) cluster and it returns “Unknown user with identity 'anonymous'.
Contact the system administrator.” Need to find out first how to access it…

Cheers Josef

From: dan young <danoyoung@gmail.com>
Reply-To: "users@nifi.apache.org" <users@nifi.apache.org>
Date: Friday, 16 November 2018 at 17:35
To: NiFi Mailing List <users@nifi.apache.org>
Subject: Re: NiFi 1.8 and stuck flowfile in Load Balanced enabled queue

Here's what we're running:

- Load Balance: Round Robin
- Compression: None
- Nodes: 3
- Are all nodes connected? : Yes
- Backpressure configuration:  Default, not changed.

Note, with our case there wasn't any other flowfiles in any queues. it was just stuck.  The
only resolution for use was a rolling restart each node in the cluster.  Since then, I have
configured the the FlowFile Expiration to 3600 seconds and we haven't seen any stuck Flowfiles....continuing
to monitor.

Regards,

Dano

On Fri, Nov 16, 2018 at 7:14 AM Mark Payne <markap14@hotmail.com<mailto:markap14@hotmail.com>>
wrote:
Hey Josef,

So a few questions to help figure out what's going on here:

- What is the Load Balance Strategy in use? (I.e., Round Robin, Partition by Attribute, etc.)
- What compression is being used (None, Compress Attributes Only, Compress Contents and Attributes)
- How many nodes in the cluster?
- Are all nodes connected?
- What is the backpressure configured to?

Is the source or destination of the Connection a Processor? If so, we can get a lot of info
by going to

http://localhost:8080/nifi-api/processors/<processor id>/diagnostics

Where http://localhost:8080/ would need to be changed to wherever your nifi instance is running
and
<processor id> is the UUID of the Processor that is either the source or destination
of the connection.
You can get the UUID of the Processor by clicking on it in the UI and then looking at the
'Operate' palette
on the left-hand side of the screen. If you can get the result of going to the URL, it will
likely be helpful,
as it shows a lot of details about the processor, as well as all of its incoming and outgoing
connections.

Thanks
-Mark





On Nov 16, 2018, at 3:05 AM, <Josef.Zahner1@swisscom.com<mailto:Josef.Zahner1@swisscom.com>>
<Josef.Zahner1@swisscom.com<mailto:Josef.Zahner1@swisscom.com>> wrote:

Hi Mark

We see the issue again, even after a fresh started cluster (where we started everything at
the same time). The files stuck for multiple seconds/minutes in the queue and the light blue
loadbalancing icon on the right side shows that it is actually loadbalancing the whole time
(even if it is just 1 or 2 files). The log (with default log levels) show no WARN or ERRORs…

Thanks in advance, Josef


From: Mark Payne <markap14@hotmail.com<mailto:markap14@hotmail.com>>
Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" <users@nifi.apache.org<mailto:users@nifi.apache.org>>
Date: Monday, 12 November 2018 at 17:19
To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" <users@nifi.apache.org<mailto:users@nifi.apache.org>>
Subject: Re: NiFi 1.8 and stuck flowfile in Load Balanced enabled queue

Hey Dan,

Have looked through the logs to see if there are any WARN or ERROR's indicating what's going
on?

Thanks
-Mark


On Nov 12, 2018, at 9:06 AM, dan young <danoyoung@gmail.com<mailto:danoyoung@gmail.com>>
wrote:

Hello,

We have two processor groups connected via the new  Load Balancing/Round Robin queue.  It
seems that a flowfile is "stuck" in this queue.  I've been watching it for some time now.
 Is there any way to trouble shoot what is stuck in the queue and why?  or maybe remove it?
 I've tried to stop the PG and empty the queue, but always says emptied 0 out of 1 flowflies...

Regards,

Dano



<Screen Shot 2018-11-12 at 7.00.37 AM.png>

Mime
View raw message