nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: PutParquet fails - "Could not rename file"
Date Fri, 01 Dec 2017 14:54:42 GMT
I think there is an open PR for a "MoveHDFS" processor that might do
what you are describing, but currently I think you'd have to use
ExecuteScript to issue an hdfs mv command.

If you are interested in trying to fix the code for PutParquet, then I
would suggest trying to add an overwrite parameter to this method:

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-hadoop-record-utils/src/main/java/org/apache/nifi/processors/hadoop/AbstractPutHDFSRecord.java#L408

Inside that method, if overwrite is true then we just need to delete
destFile before calling rename.

The value to pass in for the overwrite parameter is already available:

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-hadoop-record-utils/src/main/java/org/apache/nifi/processors/hadoop/AbstractPutHDFSRecord.java#L288



On Thu, Nov 30, 2017 at 10:14 PM, VinShar <VINAYSHARMAHK@gmail.com> wrote:
> Thanks for Reply. Actually I saw some posts where people wanted their files
> not to be overwritten by PutParquet so i thought that may be i have
> something wrong in configuration of flow.
>
> I know putParquet internally renames file on HDFS but is there a processor
> that i can use in my flow to rename a file on HDFS? i see processors to get,
> fetch, delete and put on HDFS but couldn't figure out a way to rename. If
> this is a defect then I can save file with a different name, delete existing
> file through DeleteHDFS and then rename new file to file i deleted.
> If there no processor to rename file then i will try to modify the code or
> create a new processor by extending existing one.
>
>
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Mime
View raw message