beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Kirpichov (JIRA)" <>
Subject [jira] [Closed] (BEAM-1784) DataflowPipelineJob.cancel() should be idempotent
Date Thu, 23 Mar 2017 21:39:41 GMT


Eugene Kirpichov closed BEAM-1784.
       Resolution: Fixed
    Fix Version/s: First stable release

> DataflowPipelineJob.cancel() should be idempotent
> -------------------------------------------------
>                 Key: BEAM-1784
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-dataflow
>            Reporter: Eugene Kirpichov
>            Assignee: Eugene Kirpichov
>             Fix For: First stable release
> Right now DataflowPipelineJob.cancel() issues an RPC to the Dataflow service and, if
it fails, checks whether the job is in a terminal state - in that case the failure is ignored.
> Ideally, the Dataflow-side cancel call itself would be idempotent, and would succeed
even if the job is already terminated. However, right now this is not the case.
> Additionally, Dataflow service can, for a short time, return RUNNING state even for a
job that has already been cancelled (i.e. it will reject cancel() calls, but will report state
> Because of this, if you call DataflowPipelineJob.cancel() multiple times, it might fail
(throw an exception). Calling cancel() multiple times should, logically, be ok. So, to partially
work around these issues of the Dataflow service, we can make DataflowPipelineJob.cancel()
be idempotent by itself, i.e. absorb duplicate cancel() requests. This, of course, doesn't
address the case when the job terminates externally almost concurrently to calling DataflowPipelineJob.cancel()
- but it's an improvement.

This message was sent by Atlassian JIRA

View raw message