beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-59) Switch from IOChannelFactory to FileSystems
Date Tue, 02 May 2017 06:38:04 GMT

    [ https://issues.apache.org/jira/browse/BEAM-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15992406#comment-15992406
] 

ASF GitHub Bot commented on BEAM-59:
------------------------------------

GitHub user dhalperi opened a pull request:

    https://github.com/apache/beam/pull/2818

    [BEAM-59] Delete old restrictions on output file paths

    These predate Apache Beam and are no longer relevant now that Text and Avro are implemented
    in the SDK.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dhalperi/beam remove-old-validation

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/2818.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2818
    
----
commit 50d7ff5380dc038a0797a313414fd56ab291cabb
Author: Dan Halperin <dhalperi@google.com>
Date:   2017-05-02T06:36:57Z

    [BEAM-59] Delete old restrictions on output file paths
    
    These predate Apache Beam and are no longer relevant now that Text and Avro are implemented
    in the SDK

----


> Switch from IOChannelFactory to FileSystems
> -------------------------------------------
>
>                 Key: BEAM-59
>                 URL: https://issues.apache.org/jira/browse/BEAM-59
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-core, sdk-java-gcp
>            Reporter: Daniel Halperin
>            Assignee: Daniel Halperin
>             Fix For: First stable release
>
>
> Right now, FileBasedSource and FileBasedSink communication is mediated by IOChannelFactory.
There are a number of issues:
> * Global configuration -- e.g., all 'gs://' URIs use the same credentials. This should
be per-source/per-sink/etc.
> * Supported APIs -- currently IOChannelFactory is in the "non-public API" util package
and subject to change. We need users to be able to add new backends ('s3://', 'hdfs://', etc.)
directly, without fear that they will be broken.
> * Per-backend features: e.g., creating buckets in GCS/s3, setting expiration time, etc.
> Updates:
> Design docs posted on dev@ list:
> Part 1: IOChannelFactory Redesign: 
> https://docs.google.com/document/d/11TdPyZ9_zmjokhNWM3Id-XJsVG3qel2lhdKTknmZ_7M/edit#
> Part 2: Configurable BeamFileSystem:
> https://docs.google.com/document/d/1-7vo9nLRsEEzDGnb562PuL4q9mUiq_ZVpCAiyyJw8p8/edit#heading=h.p3gc3colc2cs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message