nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juan Sequeiros <helloj...@gmail.com>
Subject Re: NiFi stops processing further flowfiles
Date Fri, 24 Nov 2017 18:59:58 GMT
Hi Panos,

The scheduling strategy defaults to timer driven on processors and I've
always avoided using event driven so if you are not purposely using it I
would just leave that setting alone. It even warns you that it is
experimental.

Your thread count is shown on the UI or can be called through RestApi
/nifi-api/flow/status

Thanks,

Juan

On Fri, Nov 24, 2017 at 1:24 PM Panos Geo <panospanos1983@outlook.com>
wrote:

> Thanks for your reply Juan.
>
>
> I have been using NiFi for almost 2 years now and didn't know about this
> setting. I 'll try to increase it and see what happens 😊
>
>
> Does it also make sense to increase the "Maximum Event Driven Thread
> Count"?
>
>
> Finally, is there a way to monitor thread usage within NiFi?
>
>
> Many thanks,
>
> Panos
>
>
>
> ------------------------------
> *From:* Juan Sequeiros <hellojuan@gmail.com>
> *Sent:* Friday, November 24, 2017 4:50 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi stops processing further flowfiles
>
> Good afternoon,
>
> My quick guess is you are running out of threads.
> If your system can handle it increase the default "max thread" count found:
>
> Left side controller settings > Maximum timer driven thread count
>
> On Fri, Nov 24, 2017 at 10:47 AM Panos Geo <panospanos1983@outlook.com>
> wrote:
>
> Hello all,
>
> We are running NiFi in a docker container that has roughly 500 processors
> running. These include consuming messages (MQTT), interaction with a
> database and some data processing with execute scripts. Our NiFi version is
> v.1.4.0 and this instance has 18GB of RAM with 12GB allocated to Java heap.
>
> Quite recently we increased the load of the incoming messages
> (1000messages/minute) that lead to more files being queued and some
> backpressure. We then have experienced what appears to be some kind of
> deadlock. NiFi stops processing any flowfiles and the processors running
> get stuck (there is a thread running constantly; we see 1 at the top right
> corner of the processor). If we try to stop these processors the icon gets
> changed to stop, but we now have 2 threads stuck at the top right corner of
> the processor. We cannot start these processors again and no other data
> flow is being processed unless we restart NiFi. We have experienced this
> multiple times over the past week.
>
> I am attaching a nifi dump in case it helps. Our nifi.properties is fairly
> standard (we just have smaller retention periods for content and
> provenance).
>
> Any thoughts? What can we do to debug this further?
>
> Many thanks,
> Panos
>
>

Mime
View raw message