beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dhalperi <>
Subject [GitHub] beam pull request #2998: [BEAM-1494] Correctly handle content-encoding in Gc...
Date Tue, 09 May 2017 19:40:35 GMT
GitHub user dhalperi opened a pull request:

    [BEAM-1494] Correctly handle content-encoding in GcsFileSystem, fixing reading of such
files in CompressedSource

    R: @jkff  thoughts?
    CC: @chamikaramj

You can merge this pull request into a Git repository by running:

    $ git pull b1494-gcs-content-encoding

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2998
commit 7ef0f8afc88b292724228fb3507e6d0c77c0b1aa
Author: Dan Halperin <>
Date:   2017-05-09T19:34:04Z

    FileBasedSource: isSplittable should not throw
    This is a legacy design from Dataflow 1.x that was a poor choice.
    All the information needed to know whether a source is splittable should
    be known at source construction time, and if runtime behavior is needed
    it should result in conservative choices, aka false.

commit 59e8e0ec27dfc498dacaaf425548681ed07a2d31
Author: Dan Halperin <>
Date:   2017-05-09T19:36:10Z

    CompressedSource: only use delegate reader if the file is splittable
    Otherwise, it's likely compressed

commit b71f5dfed5b8e56dd01cca5a71e2fa72233ab363
Author: Dan Halperin <>
Date:   2017-05-09T19:36:53Z

    GcsFileSystem: mark content-encoded files as not seekable
    That is the truth (since they are actually compressed) and will result in correct data
    when reading from them in, e.g., TextIO


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

View raw message