nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Papisetti (rpapiset)" <rpapi...@cisco.com>
Subject Re: NiFi copying windows .part files
Date Wed, 06 Dec 2017 16:56:49 GMT
File names are ending with .part (.part is getting suffixed to filename, not prefixed).

In case of files from linux file system, “.” (dot) getting prefixed while transfer is
in progress.

Thanks,
Ravi Papisetti

On 06/12/17, 1:47 AM, "Joe Witt" <joe.witt@gmail.com> wrote:

    Imagine a filename construct where you wanted to pick up any file that
    begins with the phrase 'start' but does NOT end in the phrase 'part'.
    
    The name is of a form 'begin.middle.end'.
    
    This filename start.middle.ok would get picked up.
    
    This filename start.middle.part would not.
    
    The pattern for that example would be
      start\..+\.(?!part)
    
    The key part of that is the negative lookahead for ensuring it does
    not end in part.
    
    Thanks
    
    On Wed, Dec 6, 2017 at 2:29 AM, Ravi Papisetti (rpapiset)
    <rpapiset@cisco.com> wrote:
    > Yeah..that is good idea, but we are already using this option to copy file with certain
prefix. Not sure how I can use this field to meet both exclusion and inclusion criterion.
    >
    > Any thoughts.
    >
    > Thanks,
    > Ravi Papisetti
    >
    > On 06/12/17, 1:26 AM, "Joe Witt" <joe.witt@gmail.com> wrote:
    >
    >     Ravi
    >
    >     Please use the 'File Filter' property of ListFile to control ignoring
    >     filenames until they no longer end in 'part'.
    >
    >     Thanks
    >
    >     On Wed, Dec 6, 2017 at 2:14 AM, Ravi Papisetti (rpapiset)
    >     <rpapiset@cisco.com> wrote:
    >     > Hi,
    >     >
    >     >
    >     >
    >     > We are using Apache NiFi 1.3.0
    >     >
    >     >
    >     >
    >     > We have a process flow to copy files from NFS to HDFS (with processors
    >     > ListFile, FetchFile and PutHDFS)
    >     >
    >     >
    >     >
    >     > In the NiFi process flow, ListFile is configured to listen to a directory
on
    >     > NFS. When a file (ex: x.csv) is being copied from a windows machine to NFS
    >     > (while transfer is in the middle), a part file(x.csv.part) is created at
NFS
    >     > until transfer is complete.
    >     >
    >     >
    >     >
    >     > ListFile has picked up this x.csv.part file and fetchFile picked up this
to
    >     > transfer to HDFS, didn’t update the file name back to x.csv in HDFS when
    >     > transfer is complete.
    >     >
    >     >
    >     >
    >     > But, in case a file from linux file system, while file copy to NFS is in
    >     > progress it created (.x.csv) and when transfer is complete, at both NFS
and
    >     > HDFS, filename is updated to x.csv (from .x.csv).
    >     >
    >     >
    >     >
    >     > Any thought how we can configure ListFile not to pickup these part files
or
    >     > any configurations in NiFi that fixes file names for these windows part
    >     > files?
    >     >
    >     >
    >     >
    >     > Appreciate your help.
    >     >
    >     >
    >     >
    >     > Thanks,
    >     >
    >     > Ravi Papisetti
    >
    >
    

Mime
View raw message