hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-6760) LocatedFileStatusFetcher to use listFiles(recursive)
Date Thu, 18 Aug 2016 09:23:20 GMT
Steve Loughran created MAPREDUCE-6760:
-----------------------------------------

             Summary: LocatedFileStatusFetcher to use listFiles(recursive)
                 Key: MAPREDUCE-6760
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6760
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: mrv2
    Affects Versions: 2.8.0
            Reporter: Steve Loughran


{{LocatedFileStatusFetcher }} does parallelized path listing, but it does make recursive calls
on every subdir.

If we could switch it to use {{FileSystem.listFiles(recursive)}}, object stores that have
high-performance implementations of that operation would see significant speedup.

HADOOP-13208 implements that for S3A; Azure, swift &c can do the same.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org


Mime
View raw message