nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martijn Dekkers <mart...@dekkers.org.uk>
Subject Re: NiFi 1.6.0 cluster stability with Site-to-Site
Date Fri, 10 Aug 2018 20:12:43 GMT
Whats the OS you are running on? What kind of systems? Memory stats,
network stats, JVM stats etc. How much data coming through?

On 10 August 2018 at 16:06, Joe Gresock <jgresock@gmail.com> wrote:

> Any nifi developers on this list that have any suggestions?
>
> On Wed, Aug 8, 2018 at 7:38 AM Joe Gresock <jgresock@gmail.com> wrote:
>
>> I am running a 7-node NiFi 1.6.0 cluster that performs fairly well when
>> it's simply processing its own data (putting records in Elasticsearch,
>> MongoDB, running transforms, etc.).  However, when we add receiving
>> Site-to-Site traffic to the mix, the CPU spikes to the point that the nodes
>> can't talk to each other, resulting in the inability to view or modify the
>> flow in the console.
>>
>> I have tried some basic things to mitigate this:
>> - Requested that the sending party use a comma-separated list of all 7 of
>> our nodes in their Remote Process Group that points to our cluster, in
>> hopes that that will help balance the requests
>> - Requested that the sending party use some of the batching settings on
>> the Remote Port (i.e., Count = 20, Size = 100 MB, Duration = 10 sec)
>> - Reduced the thread count on our Input Port to 2
>>
>> Are there any known nifi.properties that can be set to help mitigate this
>> problem?  Again, it only seems to be a problem when we are both receiving
>> site-to-site traffic and doing our normal processing, but taking each of
>> those activities in isolation seems to be okay.
>>
>> Thanks,
>> Joe
>>
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can
> do all this through him who gives me strength.    *-Philippians 4:12-13*
>

Mime
View raw message