nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Petronic <markpetro...@gmail.com>
Subject Can anyone explain the WHY of this PutHDFS logic?
Date Sun, 29 Nov 2015 14:41:49 GMT
This is the sort of "mystery" code that really should have some explicit
code comments. :) What are the underlying reasons for this retry logic?
This could definitely lead to a bottleneck if this loop has to run and
sleep numerous times. Just wondering what, in HDFS, results in the need to
do this? Maybe what to do to in the cluster setup/config to avoid hitting
this condition, if that is possible?

Thanks

            boolean renamed = false;
            for (int i = 0; i < 10; i++) { // try to rename multiple times.
                if (hdfs.rename(tempCopyFile, copyFile)) {
                    renamed = true;
                    break;// rename was successful
                }
                Thread.sleep(200L);// try waiting to let whatever might
cause rename failure to resolve
            }
            if (!renamed) {
                hdfs.delete(tempCopyFile, false);
                throw new ProcessException("Copied file to HDFS but could
not rename dot file " + tempCopyFile
                        + " to its final filename");
            }

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message