beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kenneth Knowles (JIRA)" <>
Subject [jira] [Commented] (BEAM-696) Side-Inputs non-deterministic with merging main-input windows
Date Tue, 04 Oct 2016 18:40:21 GMT


Kenneth Knowles commented on BEAM-696:

The SDK needs to lay out a spec, which I think is what Pei is saying, and the runner comes
up with the execution plan. To clarify - are you suggesting that {{CombineFn}} should only
be allowed side input access in {{extractOutput}}, or are you suggesting that runners be required
to wait until {{extractOutput}} _will_ be called before running a sequence of {{addInput}}*
{{mergeAccum}}* {{extractoutput}} that accesses side inputs?

The latter sounds like it could be loosed to "give a consistent view of a side input to the
sequence of {{addInput}}* {{mergeAccum}}* {{extractOutput}}" and your proposed execution plan
is one obvious choice of how to achieve it.

> Side-Inputs non-deterministic with merging main-input windows
> -------------------------------------------------------------
>                 Key: BEAM-696
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: beam-model
>            Reporter: Ben Chambers
>            Assignee: Pei He
> Side-Inputs are non-deterministic for several reasons:
> 1. Because they depend on triggering of the side-input (this is acceptable because triggers
are by their nature non-deterministic).
> 2. They depend on the current state of the main-input window in order to lookup the side-input.
This means that with merging
> 3. Any runner optimizations that affect when the side-input is looked up may cause problems
with either or both of these.
> This issue focuses on #2 -- the non-determinism of side-inputs that execute within a
Merging WindowFn.
> Possible solution would be to defer running anything that looks up the side-input until
we need to extract an output, and using the main-window at that point. Specifically, if the
main-window is a MergingWindowFn, don't execute any kind of pre-combine, instead buffer all
the inputs and combine later.
> This could still run into some non-determinism if there are triggers controlling when
we extract output.

This message was sent by Atlassian JIRA

View raw message