nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <>
Subject Re: ListSFTP Question
Date Sat, 26 Jan 2019 13:30:43 GMT
Hi Joe


Cheers Josef

From: Joe Witt <>
Reply-To: "" <>
Date: Thursday, 24 January 2019 at 17:18
To: "" <>
Subject: Re: ListSFTP Question

hey josef.  yeah we need to add a min file age property to ListSftp.  please file a jira.


On Thu, Jan 24, 2019, 11:13 AM <<>
Hi guys

We need your advice,… we use the ListSFTP processor to read files on a remote folder. The
files gets written like that:

  *   File1
  *   File2
  *   File3 (at the time of “ls” command this file is growing, we don’t know how big
it gets or when writing is finished)

So the application on that remote folder writes the files one by one, always one file grows
until a certain size (not predictable). So our issue is that NiFi reads the same file multiple
time as it grows and the modify timestamp changes. But as we don’t know when writing is
finished we don’t know how to drop all “incomplete” ListSFTP flowfiles of the “File3”
in the example.

Any help in this, in theory, pretty simple case? The best solution would be to tell the ListSFTP
that it should ignore files with a timestamp of lastmodified from at least about x minutes.
But there is no such option. We need somehow a ListSFTP delay and we shouldn’t read too
new files. But we have no idea how to do that.

Greetings Josef
View raw message