nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: absolute.path vs path for FetchFile/ListFile
Date Wed, 25 Nov 2015 19:00:08 GMT
Tony,

I would recommend that ListFile add both 'path' and 'absolute.path'. The 'path' would be relative
to the base directory being listed.
For example, if ListFile is configured to list files from /data/nifi/in and recurse subdirectories,
and it finds a file named: /data/nifi/in/123/myfile.txt
then i would expect the following attributes:

absolute.path = /data/nifi/in/123
path = ./123
filename = myfile.txt

Thanks
-Mark


> On Nov 25, 2015, at 1:56 PM, Tony Kurc <trkurc@gmail.com> wrote:
> 
> All,
> Joe and I commented on NIFI-631 that it didn't "just work" when wiring the
> processors together. ListFile was populating the attributes as
> described in CoreAttributes.java
> [1] (path being relative to the input directory, and absolute being the
> full path). FetchFile was using ${path}/${filename} as the default, which
> wouldn't grab the directory. I'm puzzled as to what the correct behavior
> should be. The description of path said it is relative ... relative to
> what? ListHDFS appears to state path is absolute [2] [3], and I expect we
> should have consistent behavior between ListHDFS and ListFile.
> 
> So, I guess I'm not sure what guidance to give on a review of NIFI-631.
> Should the default of FetchFile be changed to ${absolute.path}/${filename}
> (which may be inconsistent with other List/Fetch processor combos), or
> should ListFile be changed to have path be absolute?
> 
> [1]
> https://github.com/apache/nifi/blob/master/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/flowfile/attributes/CoreAttributes.java
> [2]
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L79
> [3]
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L442


Mime
View raw message