nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Josef.Zahn...@swisscom.com>
Subject NiFi 1.8.0 LoadBalance Strategy Issue for Connection between Funnel and FetchSFTP
Date Thu, 08 Nov 2018 07:55:34 GMT
Hi guys

We have a 8 cluster nifi cluster and do a listSFTP on the primary node. After the ListSFTP
we add some attributes and send it over a funnel to the FetchSFTP. On the connection between
the funnel and the FetchSFTP we have an “Object Threshold” of 100,some “Prioritizer”
and round robin loadbalancing to get the files in a sorted order. Right after start we had
about 800 files (expected value due to 8 nodes) in the queue between the funnel and the FetchSFTP,
but after a few hours (we get about 200k-250k files from each ListSFTP processors) the number
of files decreased to the number below. However, it seems that all nodes gets load, because
after the FetchSFTP we see a more or less even distributed load.
Next Issue or maybe misunderstanding is, that we would like to have all the listSFTP files
in a sorded order from the four folders. So we added the priority attribute where we assign
as value epoch in seconds extracted from filename. However, it seems that there is no human
understandable logic how the files get sorted in the queue between the funnel and the FetchSFTP,
because after a few hours I see files with nearly the oldest and the newest possible timestamp
in our DB (which shouldn’t be possible as we have the priority attribute with epoch time.
Is the a failure in our logic how nifi works here? Should we remove the funnel and connect
the UpdateAttribute processor directly to the FetchSFTP? Or how can we overcome the order
issue?

Thanks in advance,
Josef


[cid:image001.png@01D47740.CB955200]


[cid:image002.png@01D47740.CB955200]
Mime
View raw message