nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Papisetti (rpapiset)" <rpapi...@cisco.com>
Subject Re: NiFi copying windows .part files
Date Wed, 06 Dec 2017 07:29:40 GMT
Yeah..that is good idea, but we are already using this option to copy file with certain prefix.
Not sure how I can use this field to meet both exclusion and inclusion criterion.

Any thoughts.

Thanks,
Ravi Papisetti

On 06/12/17, 1:26 AM, "Joe Witt" <joe.witt@gmail.com> wrote:

    Ravi
    
    Please use the 'File Filter' property of ListFile to control ignoring
    filenames until they no longer end in 'part'.
    
    Thanks
    
    On Wed, Dec 6, 2017 at 2:14 AM, Ravi Papisetti (rpapiset)
    <rpapiset@cisco.com> wrote:
    > Hi,
    >
    >
    >
    > We are using Apache NiFi 1.3.0
    >
    >
    >
    > We have a process flow to copy files from NFS to HDFS (with processors
    > ListFile, FetchFile and PutHDFS)
    >
    >
    >
    > In the NiFi process flow, ListFile is configured to listen to a directory on
    > NFS. When a file (ex: x.csv) is being copied from a windows machine to NFS
    > (while transfer is in the middle), a part file(x.csv.part) is created at NFS
    > until transfer is complete.
    >
    >
    >
    > ListFile has picked up this x.csv.part file and fetchFile picked up this to
    > transfer to HDFS, didn’t update the file name back to x.csv in HDFS when
    > transfer is complete.
    >
    >
    >
    > But, in case a file from linux file system, while file copy to NFS is in
    > progress it created (.x.csv) and when transfer is complete, at both NFS and
    > HDFS, filename is updated to x.csv (from .x.csv).
    >
    >
    >
    > Any thought how we can configure ListFile not to pickup these part files or
    > any configurations in NiFi that fixes file names for these windows part
    > files?
    >
    >
    >
    > Appreciate your help.
    >
    >
    >
    > Thanks,
    >
    > Ravi Papisetti
    

Mime
View raw message