nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Gresock <jgres...@gmail.com>
Subject NiFi 1.6.0 cluster stability with Site-to-Site
Date Wed, 08 Aug 2018 11:38:55 GMT
I am running a 7-node NiFi 1.6.0 cluster that performs fairly well when
it's simply processing its own data (putting records in Elasticsearch,
MongoDB, running transforms, etc.).  However, when we add receiving
Site-to-Site traffic to the mix, the CPU spikes to the point that the nodes
can't talk to each other, resulting in the inability to view or modify the
flow in the console.

I have tried some basic things to mitigate this:
- Requested that the sending party use a comma-separated list of all 7 of
our nodes in their Remote Process Group that points to our cluster, in
hopes that that will help balance the requests
- Requested that the sending party use some of the batching settings on the
Remote Port (i.e., Count = 20, Size = 100 MB, Duration = 10 sec)
- Reduced the thread count on our Input Port to 2

Are there any known nifi.properties that can be set to help mitigate this
problem?  Again, it only seems to be a problem when we are both receiving
site-to-site traffic and doing our normal processing, but taking each of
those activities in isolation seems to be okay.

Thanks,
Joe

Mime
View raw message