nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tony Kurc <trk...@gmail.com>
Subject Re: absolute.path vs path for FetchFile/ListFile
Date Wed, 25 Nov 2015 19:12:19 GMT
I am 100% in favor of keeping the relative path (I brought up out of band
the value if the Lister and Fetcher were different machines with different
mount points). I think is just a matter of what attribute to fill with what
value.


On Wed, Nov 25, 2015 at 2:09 PM, Joe Skora <jskora@gmail.com> wrote:

> Mark,
>
> What you described is the behavior of ListFile (in spite of confusing doc
> info).
>
> JoeW,
>
> Consistency with ListHDFS makes sense,  and if that is the desired behavior
> it's easy to change ListFile.  But CoreAttributres state "The flowfile's
> path indicates the relative directory" and if that's not true, does
> CoreAttributes need revision too?
>
> Thanks,
> Joe
>
> On Wed, Nov 25, 2015 at 2:00 PM, Mark Payne <markap14@hotmail.com> wrote:
>
> > Tony,
> >
> > I would recommend that ListFile add both 'path' and 'absolute.path'. The
> > 'path' would be relative to the base directory being listed.
> > For example, if ListFile is configured to list files from /data/nifi/in
> > and recurse subdirectories, and it finds a file named:
> > /data/nifi/in/123/myfile.txt
> > then i would expect the following attributes:
> >
> > absolute.path = /data/nifi/in/123
> > path = ./123
> > filename = myfile.txt
> >
> > Thanks
> > -Mark
> >
> >
> > > On Nov 25, 2015, at 1:56 PM, Tony Kurc <trkurc@gmail.com> wrote:
> > >
> > > All,
> > > Joe and I commented on NIFI-631 that it didn't "just work" when wiring
> > the
> > > processors together. ListFile was populating the attributes as
> > > described in CoreAttributes.java
> > > [1] (path being relative to the input directory, and absolute being the
> > > full path). FetchFile was using ${path}/${filename} as the default,
> which
> > > wouldn't grab the directory. I'm puzzled as to what the correct
> behavior
> > > should be. The description of path said it is relative ... relative to
> > > what? ListHDFS appears to state path is absolute [2] [3], and I expect
> we
> > > should have consistent behavior between ListHDFS and ListFile.
> > >
> > > So, I guess I'm not sure what guidance to give on a review of NIFI-631.
> > > Should the default of FetchFile be changed to
> > ${absolute.path}/${filename}
> > > (which may be inconsistent with other List/Fetch processor combos), or
> > > should ListFile be changed to have path be absolute?
> > >
> > > [1]
> > >
> >
> https://github.com/apache/nifi/blob/master/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/flowfile/attributes/CoreAttributes.java
> > > [2]
> > >
> >
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L79
> > > [3]
> > >
> >
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L442
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message