beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-1630) Add Splittable DoFn to Python SDK
Date Thu, 08 Jun 2017 22:08:18 GMT

    [ https://issues.apache.org/jira/browse/BEAM-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043527#comment-16043527
] 

ASF GitHub Bot commented on BEAM-1630:
--------------------------------------

GitHub user chamikaramj opened a pull request:

    https://github.com/apache/beam/pull/3333

    [BEAM-1630] Adds ability to dynamically replace PTransforms during runtime.

    Adds two new interfaces, PTransformMatcher and PTransformOverride.
    
    Currently only supports replacements where input and output types are an exact match (we
have to address complexities due to type hints before supporting replacements with different
types).
    
    This can be used to dynamically update a populated pipeline at runtime. Each runner can
configure it's own overrides.
    
    This will be used by SplittableDoFn where matching ParDo transforms will be dynamically
replaced by SplittableParDo.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/chamikaramj/beam sdf_direct_runner_ptransform_override

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/3333.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3333
    
----

----


> Add Splittable DoFn to Python SDK
> ---------------------------------
>
>                 Key: BEAM-1630
>                 URL: https://issues.apache.org/jira/browse/BEAM-1630
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-py
>            Reporter: Chamikara Jayalath
>            Assignee: Chamikara Jayalath
>
> Splittable DoFn [1] is currently being implemented for Java SDK [2]. We should add this
to Python SDK as well.
> Following document proposes an API for this.
> https://docs.google.com/document/d/1h_zprJrOilivK2xfvl4L42vaX4DMYGfH1YDmi-s_ozM/edit?usp=sharing
> [1] https://s.apache.org/splittable-do-fn
> [2] https://lists.apache.org/thread.html/0ce61ac162460a149d5c93cdface37cc383f8030fe86ca09e5699b18@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message