beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (BEAM-5417) FileSystems.match behaviour diff between GCS and local file system
Date Fri, 21 Sep 2018 17:06:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-5417?focusedWorklogId=146469&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146469
]

ASF GitHub Bot logged work on BEAM-5417:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 21/Sep/18 17:05
            Start Date: 21/Sep/18 17:05
    Worklog Time Spent: 10m 
      Work Description: udim commented on issue #6423: [BEAM-5417] Parity between GCS and
local match
URL: https://github.com/apache/beam/pull/6423#issuecomment-423607027
 
 
   @joar 
   I see two options here:
   - Convert the glob into a regex instead. Something like: `re.sub(r'\*', r'[^/\\\\]*', pattern)`,
but it's more complicated because of special characters that need escaping. See how it's done
in Java SDK: https://github.com/apache/beam/blob/279a05604b83a54e8e5a79e13d8761f94841f326/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java#L155
   - Use Python 3's pathlib (PurePath.match()), falling back on [pathlib2](https://pypi.org/project/pathlib2/)
in Python 2.
   
   I'm leaning towards the second option, since it's less code to maintain and uses standard
libraries in Python 3.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 146469)
    Time Spent: 1h 10m  (was: 1h)

> FileSystems.match behaviour diff between GCS and local file system
> ------------------------------------------------------------------
>
>                 Key: BEAM-5417
>                 URL: https://issues.apache.org/jira/browse/BEAM-5417
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py-core
>    Affects Versions: 2.5.0, 2.6.0
>            Reporter: Joar Wandborg
>            Assignee: Chamikara Jayalath
>            Priority: Major
>          Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Given the directory structure:
>  
> {noformat}
> .
> ├── filesystem-match-test
> │   ├── a
> │   │   └── file.txt
> │   └── b
> │       └── file.txt
> └── filesystem-match-test.py
> {noformat}
>  
> Where {{filesystem-match-test.py}} contains:
> {code:python}
> from __future__ import print_function
> import os
> import posixpath
> from apache_beam.io.filesystem import MatchResult
> from apache_beam.io.filesystems import FileSystems
> BASES = [
>     os.path.join(os.path.dirname(__file__), "./"),
>     "gs://my-bucket/test/",
> ]
> pattern = "filesystem-match-test/*/file.txt"
> for base_path in BASES:
>     full_pattern = posixpath.join(base_path, pattern)
>     print("full_pattern: {}".format(full_pattern))
>     match_result = FileSystems.match([full_pattern])[0]  # type: MatchResult
>     print("metadata list: {}".format(match_result.metadata_list))
> {code}
> Running {{python filesystem-match-test.py}} does not match any files locally, but does
match files on GCS:
> {noformat}
> full_pattern: ./filesystem-match-test/*/file.txt
> metadata list: []
> full_pattern: gs://my-bucket/test/filesystem-match-test/*/file.txt
> metadata list: [FileMetadata(gs://my-bucket/test/filesystem-match-test/a/file.txt, 6),
FileMetadata(gs://my-bucket/test/filesystem-match-test/b/file.txt, 6)]
> {noformat}
> The expected result is that a/file.txt and b/file.txt should be matched for both patterns.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message