nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 진유리 <>
Subject What kind of way do you recommend for the load balancing? (RPG/HTTP processors)
Date Tue, 20 Jun 2017 02:38:53 GMT
Hi All,
I have some questions about load balancing for clustered NiFi v1.0.0 (2

I consider 2 ways.

*1) RPG way : Remote Processor Group (HTTP/RAW) + InputPort*
*2) HTTP way : PostHTTP + ListenHTTP*

*Could you tell me which one is better and explain why ?*
On my test case, HTTP way is faster than RPG way but HTTP way have a
disadvantage to assign unique port number for each ListenHTTP processor.
(Actually, I don't understand why HTTP way is faster than  RPG way..)

Moreover, I found some strange things on my workflow.
This in my NiFi workflow to compare performance between PRG way and HTTP way

GenerateFlowFile(On Primary Node) -> DistributeLoad -> RPG (to node 1)

            -> RPG (to node 2)
DistributeLoad -> PostHTTP (to node 1)

            -> PostHTTP (to node 2)
InputPort     -> PutFile
ListenHTTP-> PutFile

First, I got Socket Exception on 'PostHTTP' processor.
( Connection reset, Broken pipe (Write failed))
I guess  it cause that calling this too many times.

*PostHTTP processor shows error mark and logs but RPG does not show
*I guess both have same problem because both don't work anymore.*

*And I select 'Round Robin' Strategy all 'DistributeLoad' Processors.*
*But result of above 2 ways is different for each node.*

- RPG way : One node wrote 2 times more files than another node
- HTTP way : Each node wrote almost same number files

*Please share your opinions or tips for load balancing.*

-Yuri Jin-

View raw message