nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charlie Meyer <charlie.me...@civitaslearning.com>
Subject Re: NiFi Remote Process Group Performance Tuning
Date Wed, 21 Nov 2018 04:48:46 GMT
Thanks Koji,

That helped a lot!

On Tue, Nov 20, 2018 at 6:48 PM Koji Kawamura <ijokarumawak@gmail.com>
wrote:

> Hi Charlie,
>
> How about increasing the batch configuration?
> You can configure batch settings at the sender side by:
> 1. Disable transmission
> 2. Right clicking the RPG, select 'Manage Remote Ports'
> 3. Click the pencil icon to edit remote port configuration
> 4. Specify Batch settings, such as 'Count' to 1000
>
> Thanks,
> Koji
>
> On Tue, Nov 20, 2018 at 12:14 PM Charlie Meyer <
> charlie.meyer@civitaslearning.com> wrote:
>
>> Hi,
>>
>> I have a setup with 2 nifi deployments that communicate via HTTP remote
>> process groups.
>>
>> I have noticed that often, flowfiles will backpressure buffer on the
>> sender side of the RPG.  Here is an example of what I see:
>>
>> [image: image.png]
>>
>> The incoming connection to the RPG is configured with a limit of 500k
>> flow files in this example. I only see 1 "thread" running for the sender
>> RPG. Looking at the receiving input port that this RPG is communicating
>> with, I have it configured like:
>>
>> [image: image.png]
>> I have bumped up the number of threads allocated to the input port to 4,
>> but yet the connection out of the input port to the rest of the flow is not
>> backing up.
>>
>> Are there things that can be tuned or other RPG best practices that can
>> be applied to the sender side of the RPG to increase throughput?
>>
>> Thanks!
>>
>> -Charlie
>>
>

Mime
View raw message