beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (BEAM-2859) ProcessingTime based timers are not properly fired in case the watermark stays put
Date Tue, 12 Sep 2017 07:40:00 GMT


ASF GitHub Bot commented on BEAM-2859:

GitHub user staslev opened a pull request:

    [BEAM-2859] Fixed processing timers not being properly fired

    Follow this checklist to help us incorporate your contribution quickly and easily:
     - [ ] Make sure there is a [JIRA issue](
filed for the change (usually before you start working on it).  Trivial changes like typos
do not require a JIRA issue.  Your pull request should address just this issue, without pulling
in other changes.
     - [ ] Each commit in the pull request should have a meaningful subject line and body.
     - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`,
where you replace `BEAM-XXX` with the appropriate JIRA issue.
     - [ ] Write a pull request description that is detailed enough to understand what the
pull request does, how, and why.
     - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will
be performed on your pull request automatically.
     - [ ] If this contribution is large, please file an Apache [Individual Contributor License

You can merge this pull request into a Git repository by running:

    $ git pull BEAM-2859-ProcessingTime-based-timers-are-not-properly-fired

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3840
commit afe9bda0469d189d61cfdb58123629910f816820
Author: Stas Levin <>
Date:   2017-09-12T07:34:45Z

    [BEAM-2859] Fixed processing timers not being properly fired when watermark stays put
by tweaking the way spark-runner was delivering timers to reduceFnRunner in SparkGroupAlsoByWindowViaWindowSet


> ProcessingTime based timers are not properly fired in case the watermark stays put
> ----------------------------------------------------------------------------------
>                 Key: BEAM-2859
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-spark
>    Affects Versions: 2.0.0, 2.1.0
>            Reporter: Stas Levin
>            Assignee: Stas Levin
> {{AfterProcessingTime}} based timers are not fired when the input watermark does not
advance, preventing from buffered element to be emitted.
> The reason seems to be that {{SparkTimerInternals#getTimersReadyToProcess()}} determines
what triggers are ready to be processed based on the following condition: 
> {code:java}
> timer.getTimestamp().isBefore(inputWatermark)
> {code}
> However, if the timer domain is {{TimeDomain.PROCESSING_TIME}} the position of the input
watermark should *NOT* have effect.
> In addition, {{SparkTimerInternals#getTimersReadyToProcess()}} deletes timers once they
are deemed eligible for processing (but will not necessarily fire). 
> This may not be the correct behavior for timers in general and for timers in the {{TimeDomain.PROCESSING_TIME}}
in particular, since they should remain scheduled until the corresponding window expires and
all state is cleared.
> For instance, consider a timer that is found eligible for processing and is thus deleted,
then it just so happens to be that its {{shouldFire()}} returns {{false}} and it is not fired
and needs to be re-run next time around, but won't, since it's been deleted. The implied moral
being that _"eligible for processing"_ does not imply _"should be deleted"_.
> It may be better to avoid removing timers in {{SparkTimerInternals#getTimersReadyToProcess()}}
and leave timer management up to {{ReduceFnRunner#clearAllState()}} which has more context
to determine whether it's time for a given timer to be deleted.

This message was sent by Atlassian JIRA

View raw message