hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Work logged] (HADOOP-17281) Implement FileSystem.listStatusIterator() in S3AFileSystem
Date Tue, 06 Oct 2020 13:00:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-17281?focusedWorklogId=495884&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-495884
]

ASF GitHub Bot logged work on HADOOP-17281:
-------------------------------------------

                Author: ASF GitHub Bot
            Created on: 06/Oct/20 12:59
            Start Date: 06/Oct/20 12:59
    Worklog Time Spent: 10m 
      Work Description: steveloughran commented on a change in pull request #2354:
URL: https://github.com/apache/hadoop/pull/2354#discussion_r500254718



##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java
##########
@@ -1520,7 +1521,8 @@ public static TreeScanResults treeWalk(FileSystem fs, Path path)
       while (true) {
         list.add(iterator.next());
       }
-    } catch (NoSuchElementException expected) {
+    } catch (NoSuchElementException | IllegalStateException expected) {

Review comment:
       as discussed: better to fix DirListingIterator to match RemoteIterator API; we can
consider it's failure a bug

##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractGetFileStatusTest.java
##########
@@ -359,16 +357,39 @@ public void testListStatusFile() throws Throwable {
   public void testListStatusIteratorFile() throws Throwable {
     describe("test the listStatusIterator(path) on a file");
     Path f = touchf("listStItrFile");
+
     List<FileStatus> statusList = (List<FileStatus>) iteratorToList(
             getFileSystem().listStatusIterator(f));
-    assertEquals("size of file list returned", 1, statusList.size());
-    assertIsNamedFile(f, statusList.get(0));
+    validateListingForFile(f, statusList, false);
+
     List<FileStatus> statusList2 =
             (List<FileStatus>) iteratorToListThroughNextCallsAlone(
                     getFileSystem().listStatusIterator(f));
-    assertEquals("size of file list returned through next() calls",
-            1, statusList2.size());
-    assertIsNamedFile(f, statusList2.get(0));
+    validateListingForFile(f, statusList2, true);
+  }
+
+  /**
+   * Validate listing result for an input path which is file.
+   * @param f file.
+   * @param statusList list status of a file.
+   * @param nextCallAlone whether the listing generated just using
+   *                      next() calls.
+   */
+  private void validateListingForFile(Path f,
+                                      List<FileStatus> statusList,
+                                      boolean nextCallAlone) {
+    String msg = String.format("size of file list returned using %s should " +
+            "be 1", nextCallAlone ?
+            "next() calls alone" : "hasNext() and next() calls");
+    Assertions.assertThat(statusList)
+            .describedAs(msg)
+            .hasSize(1);
+    Assertions.assertThat(statusList.get(0).getPath().toString())
+            .describedAs("path returned should match with the input path")

Review comment:
       any reason not to leave both as Path and let Path.equals() to do the work? It compares
URIs




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 495884)
    Time Spent: 1.5h  (was: 1h 20m)

> Implement FileSystem.listStatusIterator() in S3AFileSystem
> ----------------------------------------------------------
>
>                 Key: HADOOP-17281
>                 URL: https://issues.apache.org/jira/browse/HADOOP-17281
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: fs/s3
>    Affects Versions: 3.3.0
>            Reporter: Mukund Thakur
>            Assignee: Mukund Thakur
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Currently S3AFileSystem only implements listStatus() api which returns an array. Once
we implement the listStatusIterator(), clients can benefit from the async listing done recently 
> https://issues.apache.org/jira/browse/HADOOP-17074  by performing some tasks on files
while iterating them.
>  
> CC [~stevel]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message