beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (BEAM-115) Beam Runner API
Date Fri, 26 May 2017 06:56:04 GMT


ASF GitHub Bot commented on BEAM-115:

GitHub user kennknowles opened a pull request:

    [BEAM-115] Runner API Translations for StateSpec and TimerSpec

    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
     - [x] Make sure the PR title is formatted like:
       `[BEAM-<Jira issue #>] Description of pull request`
     - [ ] Make sure tests pass via `mvn clean verify`.
     - [x] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [x] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](
    R: @tgroh 
    The awkward bit is that a combining `StateSpec` is statically typed to take a `CombineFn`.
So when we get around to issuing Fn State API requests, we'll need to work around that.

You can merge this pull request into a Git repository by running:

    $ git pull translate-StateSpec

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3233
commit c87b72d5522e3369becd7fbe022824f4f223a9ae
Author: Kenneth Knowles <>
Date:   2017-05-25T14:12:08Z

    Flesh out TimerSpec and StateSpec in Runner API

commit 55af992a3a465b333bc3ae51262bfc43474d90e8
Author: Kenneth Knowles <>
Date:   2017-05-25T14:25:08Z

    Mark CombineFnWithContext StateSpecs internal

commit 34eca25c5a6b6f3733f51fb6cb421cc755b7058c
Author: Kenneth Knowles <>
Date:   2017-05-25T14:27:52Z

    Add case dispatch to StateSpec
    This is different than a StateBinder: for a binder, the id is needed and
    the StateSpec controls the return type. For case dispatch, the
    dispatcher controls the type and it should just be reading the spec,
    which does not require the id. Eventually, StateBinder could be removed
    in favor of StateSpec.Cases<Function<String, StateT>>.

commit 09aeab25a92ef961a8968a5a3e863786750dff46
Author: Kenneth Knowles <>
Date:   2017-05-25T20:02:15Z

    Allow translation to throw IOException

commit 9eb1ef070506e7b419bd388f2a0f9407056a8bcb
Author: Kenneth Knowles <>
Date:   2017-05-26T05:51:18Z

    Make Java serialized CombineFn URN public

commit 5aa01d64ff50d9396137ada84b281700ce1d8d8d
Author: Kenneth Knowles <>
Date:   2017-05-25T14:12:29Z

    Implement TimerSpec and StateSpec translation


> Beam Runner API
> ---------------
>                 Key: BEAM-115
>                 URL:
>             Project: Beam
>          Issue Type: Improvement
>          Components: beam-model-runner-api
>            Reporter: Kenneth Knowles
>            Assignee: Kenneth Knowles
> The PipelineRunner API from the SDK is not ideal for the Beam technical vision.
> It has technical limitations:
>  - The user's DAG (even including library expansions) is never explicitly represented,
so it cannot be analyzed except incrementally, and cannot necessarily be reconstructed (for
example, to display it!).
>  - The flattened DAG of just primitive transforms isn't well-suited for display or transform
>  - The TransformHierarchy isn't well-suited for optimizations.
>  - The user must realistically pre-commit to a runner, and its configuration (batch vs
streaming) prior to graph construction, since the runner will be modifying the graph as it
is built.
>  - It is fairly language- and SDK-specific.
> It has usability issues (these are not from intuition, but derived from actual cases
of failure to use according to the design)
>  - The interleaving of apply() methods in PTransform/Pipeline/PipelineRunner is confusing.
>  - The TransformHierarchy, accessible only via visitor traversals, is cumbersome.
>  - The staging of construction-time vs run-time is not always obvious.
> These are just examples. This ticket tracks designing, coming to consensus, and building
an API that more simply and directly supports the technical vision.

This message was sent by Atlassian JIRA

View raw message