beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-360) Add a framework for creating Python-SDK sources for new file types
Date Fri, 08 Jul 2016 23:32:11 GMT

    [ https://issues.apache.org/jira/browse/BEAM-360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368706#comment-15368706
] 

ASF GitHub Bot commented on BEAM-360:
-------------------------------------

GitHub user chamikaramj reopened a pull request:

    https://github.com/apache/incubator-beam/pull/599

    [BEAM-360] Some updates related to dynamic work rebalancing of custom sources.

    Adds a class 'iobase.BoundedSourceSplit' to represent dynamic work rebalancing results
of custom sources.
    
    Updates Dataflow runner specific code (apiclient.py) to support dynamic work rebalancing
custom sources.
    
    Updates 'OffsetRangeTracker' so that the result of 'position_at_fraction()'' is a 'long'
instead of a 'float'.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/chamikaramj/incubator-beam custom_sources_dwr

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-beam/pull/599.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #599
    
----
commit 19a41ccf5bcf00192e3646258eae0cbce85da23b
Author: Chamikara Jayalath <chamikara@apache.org>
Date:   2016-07-07T03:25:04Z

    Adds a class 'iobase.BoundedSourceSplit' to represent dynamic work rebalancing result
of custom sources.
    
    Updates Dataflow runner specific code (apiclient.py) to support dynamic work rebalancing
custom sources.
    
    Updates 'OffsetRangeTracker' so that the result of 'position_at_fraction()'' is a 'long'
instead of a 'float'.

commit 4415989ef0dfd656643e6e8575b6e2090b4437b5
Author: Chamikara Jayalath <chamikara@apache.org>
Date:   2016-07-07T03:34:21Z

    Adds more comments.

commit 6aa697465e88f827a3121a1de8bad1b810d904da
Author: Chamikara Jayalath <chamikara@google.com>
Date:   2016-07-07T04:41:20Z

    Some updates related to dynamic work rebalancing custom sources.
    
    Adds a class 'iobase.BoundedSourceSplit' to represent dynamic work rebalancing result
of custom sources.
    
    Updates Dataflow runner specific code (apiclient.py) to support dynamic work rebalancing
custom sources.
    
    Updates 'OffsetRangeTracker' so that the result of 'position_at_fraction()'' is a 'long'
instead of a 'float'.

commit 1e01b1f5cd70e5b39cd064577110898c623e524a
Author: Chamikara Jayalath <chamikara@google.com>
Date:   2016-07-08T19:01:42Z

    Reverting some updates.

commit 171df1ec3333edd51c7c72db309d526dfa9badf1
Author: Chamikara Jayalath <chamikara@google.com>
Date:   2016-07-08T22:34:52Z

    Adds a method 'fileio.ChannelFactory.size_in_bytes()'' that can be used to determine the
size of a single file.
    
    Updates 'filebasedsource' to use this method when determining size of files.

----


> Add a framework for creating Python-SDK sources for new file types
> ------------------------------------------------------------------
>
>                 Key: BEAM-360
>                 URL: https://issues.apache.org/jira/browse/BEAM-360
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-py
>            Reporter: Chamikara Jayalath
>            Assignee: Chamikara Jayalath
>
> We already have a framework for creating new sources for Beam Python SDK - https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/iobase.py#L326
> It would be great if we can add a framework on top of this that encapsulates logic common
to sources that are based on files. This framework can include following features that are
common to sources based on files.
> (1) glob expansion
> (2) support for new file-systems
> (3) dynamic work rebalancing based on byte offsets
> (4) support for reading compressed files.
> Java SDK has a similar framework and it's available at - https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSource.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message