nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Taft <a...@adamtaft.com>
Subject Re: Keep Files
Date Mon, 16 Nov 2015 15:50:10 GMT
Oooh, neat idea Salvatore.  +1 to creativity.  Really interesting.

Adam

On Mon, Nov 16, 2015 at 6:25 AM, Salvatore Papa <salvatore.papa@gmail.com>
wrote:

> If you're on a linux system, a alternative i've used in the past is to
> create another directory, full of symlinks pointing to the original
> directory.
>
> As an example, assuming you have a directory: /data/input_files/ full of
> files, create a directory /data/input_links/, and from that new directory,
> do: "ln -s ../input_files/* ./"
>
> Now in NiFi, use the original GetFile processor, configured with
> /data/input_links/, and set Keep Source File to False. When the GetFile
> processor picks up the file, it'll read the contents and create a flowfile
> by following the symlink, delete the symlink, and the original file will
> remain in /data/input_files.
>
> On Mon, Nov 16, 2015 at 12:00 PM, Adam Taft <adam@adamtaft.com> wrote:
>
> > Also, as a potential work-around, it's possible to use GetFile with
> > "delete" mode and then somewhere in your flow, use PutFile to place the
> > file back down into a "complete" directory.  i.e. something like:
> >
> > /path/incoming  <- use GetFile to pick up files here
> > /path/complete  <- use PutFile to place files here after processing
> >
> > As a variation of the above, if you need the files consistently in the
> same
> > directory, you could configure GetFile to only pick up certain file
> > patterns.  In this way, you could rename a file after it has been
> > processed:
> >
> > /path/incoming  <- use GetFile to pick up files named $filename.new
> > /path/incoming  <- rename file (using UpdateAttribute) to
> > $filename.complete and use PutFile to place files here after rename
> >
> > Hope that gives you some possible alternatives.
> >
> > Adam
> >
> >
> >
> > On Sat, Nov 14, 2015 at 10:49 AM, Mark Petronic <markpetronic@gmail.com>
> > wrote:
> >
> > > Keep, yes, There is a parameter to configure that. Read once. No. But
> > there
> > > is a set of processors in the works to address that. ListFile and
> > > FetchFile. ListFile will return the list of files that have changed
> since
> > > the last time the files were read - it is stateful. FetchFile can then
> > take
> > > a list and fetch them, and I would assume it would have a parameter for
> > > keep=<yes|no> like GetFile. Not sure of the status of the changes -
> have
> > > not checked recently but see:
> > > https://issues.apache.org/jira/browse/NIFI-631
> > >
> > > Mark
> > >
> > > On Fri, Nov 13, 2015 at 8:55 AM, plj <plj@mitre.org> wrote:
> > >
> > > > Is there a way for GetFile to not delete a file but only read it
> > once?  I
> > > > have a directory with files in it.  I only want the new files that
> are
> > > > added
> > > > to the to be processed.  It seems that if I set GetFile to not delete
> > the
> > > > files, the same files get read over and over.
> > > >
> > > >
> > > > thoughts?
> > > >
> > > >
> > > >
> > > > --
> > > > View this message in context:
> > > >
> > >
> >
> http://apache-nifi-developer-list.39713.n7.nabble.com/Keep-Files-tp4864.html
> > > > Sent from the Apache NiFi Developer List mailing list archive at
> > > > Nabble.com.
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message