beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (BEAM-4711) LocalFileSystem.delete doesn't support globbing
Date Tue, 18 Sep 2018 20:52:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-4711?focusedWorklogId=145468&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145468
]

ASF GitHub Bot logged work on BEAM-4711:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 18/Sep/18 20:51
            Start Date: 18/Sep/18 20:51
    Worklog Time Spent: 10m 
      Work Description: lukecwik commented on a change in pull request #5863: [BEAM-4711]
fix globbing in LocalFileSystem.delete
URL: https://github.com/apache/beam/pull/5863#discussion_r218590309
 
 

 ##########
 File path: sdks/python/apache_beam/io/localfilesystem.py
 ##########
 @@ -321,11 +321,22 @@ def _delete_path(path):
         raise IOError(err)
 
     exceptions = {}
-    for path in paths:
+
+    def try_delete(path):
       try:
         _delete_path(path)
       except Exception as e:  # pylint: disable=broad-except
         exceptions[path] = e
 
+    for match_result in self.match(paths):
+      metadata_list = match_result.metadata_list
+
+      if not metadata_list:
 
 Review comment:
   Turns out that we test that delete throws an error if non existent and that we have an
inconsistent definition between multiple implementations (e.g Java LocalFileSystem). Filed
https://issues.apache.org/jira/browse/BEAM-5425 to ask whether the contract should be changed
to allow prior partial successes to continue and for errors to not be thrown for non-existent
files.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 145468)
    Time Spent: 1h 40m  (was: 1.5h)

> LocalFileSystem.delete doesn't support globbing
> -----------------------------------------------
>
>                 Key: BEAM-4711
>                 URL: https://issues.apache.org/jira/browse/BEAM-4711
>             Project: Beam
>          Issue Type: Task
>          Components: sdk-py-core
>    Affects Versions: 2.5.0
>            Reporter: Ryan Williams
>            Assignee: Ryan Williams
>            Priority: Minor
>          Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> I attempted to run {{wordcount_it_test:WordCountIT.test_wordcount_it}} locally with {{DirectRunner}}:
> {code}
> python setup.py nosetests \
>   --tests apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it \
>   --test-pipeline-options="--output=foo"
> {code}
> It failed in [the {{delete_files}} cleanup command|https://github.com/apache/beam/blob/a58f1ffaafb0e2ebcc73a1c5abfb05a15ec6a84b/sdks/python/apache_beam/examples/wordcount_it_test.py#L64]:
> {code}
> root: WARNING: Retry with exponential backoff: waiting for 11.1454450937 seconds before
retrying delete_files because we caught exception: BeamIOError: Delete operation failed with
exceptions {'foo/1530557644/results*': IOError(OSError(2, 'No such file or directory'),)}
>  Traceback for above exception (most recent call last):
>   File "/Users/ryan/c/beam/sdks/python/apache_beam/utils/retry.py", line 184, in wrapper
>     return fun(*args, **kwargs)
>   File "/Users/ryan/c/beam/sdks/python/apache_beam/testing/test_utils.py", line 136,
in delete_files
>     FileSystems.delete(file_paths)
>   File "/Users/ryan/c/beam/sdks/python/apache_beam/io/filesystems.py", line 282, in delete
>     return filesystem.delete(paths)
>   File "/Users/ryan/c/beam/sdks/python/apache_beam/io/localfilesystem.py", line 304,
in delete
>     raise BeamIOError("Delete operation failed", exceptions)
> {code}
> The line:
> {code}
> self.addCleanup(delete_files, [output + '*'])
> {code}
> works as expected in GCS, and deletes a test's output-directory, but it fails in on the
local-filesystem, which doesn't expand globs before attempting to delete paths.
> It would be good to make these consistent, presumably by adding glob-support to {{LocalFileSystem}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message