beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (BEAM-2447) Reintroduce DoFn.ProcessContinuation
Date Wed, 14 Jun 2017 22:01:00 GMT


ASF GitHub Bot commented on BEAM-2447:

GitHub user jkff opened a pull request:

    [BEAM-2447] Reintroduces DoFn.ProcessContinuation

    This is a revert of, amended according to the
new semantics described in
    This will require performing a traditional Dataflow worker dance beforehand, to change
the signature of DoFnInvoker. I'll do that in a separate surgical PR to be submitted before
this one, but the current PR is ready for review anyway.
    R: @tgroh 

You can merge this pull request into a Git repository by running:

    $ git pull process-cont

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3360
commit 0c1d725d2a1bf490ae7859c6906d7f74e3b8ffdd
Author: Eugene Kirpichov <>
Date:   2017-06-13T23:50:35Z

    [BEAM-2447] Reintroduces DoFn.ProcessContinuation


> Reintroduce DoFn.ProcessContinuation
> ------------------------------------
>                 Key: BEAM-2447
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-core
>            Reporter: Eugene Kirpichov
>            Assignee: Eugene Kirpichov
> ProcessContinuation.resume() is useful for tailing files - when we reach current EOF,
we want to voluntarily suspend the process() call rather than wait for runner to checkpoint
> In BEAM-1903, DoFn.ProcessContinuation was removed because there was ambiguity about
the semantics of resume() especially w.r.t. the following situation described in
: the runner has taken a checkpoint on the tracker, and then the ProcessElement call returns
resume() signaling that the work is still not done - then there's 2 checkpoints to deal with.
> Instead, the proper way to refine this semantics is:
> - After checkpoint() on a RestrictionTracker, the tracker MUST fail all subsequent tryClaim()
calls, and MUST succeed in checkDone().
> - After a failed tryClaim() call, the ProcessElement method MUST return stop()
> - So ProcessElement can return resume() only *instead* of doing tryClaim()
> - Then, if the runner has already taken a checkpoint but tracker has returned resume(),
we do not need to take a new checkpoint - the one already taken already accurately describes
the remainder of the work.

This message was sent by Atlassian JIRA

View raw message