nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: ListSFTP Question
Date Thu, 24 Jan 2019 16:18:01 GMT
hey josef.  yeah we need to add a min file age property to ListSftp.
please file a jira.

thanks

On Thu, Jan 24, 2019, 11:13 AM <Josef.Zahner1@swisscom.com wrote:

> Hi guys
>
>
>
> We need your advice,… we use the ListSFTP processor to read files on a
> remote folder. The files gets written like that:
>
>
>
>    - File1
>    - File2
>    - File3 (at the time of “ls” command this file is growing, we don’t
>    know how big it gets or when writing is finished)
>
>
>
> So the application on that remote folder writes the files one by one,
> always one file grows until a certain size (not predictable). So our issue
> is that NiFi reads the same file multiple time as it grows and the modify
> timestamp changes. But as we don’t know when writing is finished we don’t
> know how to drop all “incomplete” ListSFTP flowfiles of the “File3” in the
> example.
>
>
>
> Any help in this, in theory, pretty simple case? The best solution would
> be to tell the ListSFTP that it should ignore files with a timestamp of
> lastmodified from at least about x minutes. But there is no such option. We
> need somehow a ListSFTP delay and we shouldn’t read too new files. But we
> have no idea how to do that.
>
>
>
> Greetings Josef
>

Mime
View raw message