beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pei He (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-101) Data-driven triggers
Date Thu, 01 Jun 2017 13:09:04 GMT

    [ https://issues.apache.org/jira/browse/BEAM-101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16032951#comment-16032951
] 

Pei He commented on BEAM-101:
-----------------------------

Background:
We need a way to define a custom trigger, such as AfterPaneTriggerWithMaxInterval (trigger
after certain element count or when the time gap between two elements exceeds the maxInterval).


Currently, there are fixed number of trigger cases, and the only way to create custom trigger
is to modify existing one on runners/core:
https://github.com/apache/beam/compare/master...peihe:custom-after-pane?expand=1

1. Could we make the AfterPane trigger customizable based on more information, such as timestamp?
2. Could we define the data-driven trigger based on a CombineFn<ElementT, AccumT, Boolean>
(or a equivalence, such as the following TriggerFn)?
class TriggerFn<ElementT, AccumT> implements Serializable {
  public abstract AccumT createAccumulator();
  public abstract AccumT onElement(ElementT element, BoundedWindow window, Instant timestamp);
  public abstract boolean shouldFire(AccumT accum);
}


> Data-driven triggers
> --------------------
>
>                 Key: BEAM-101
>                 URL: https://issues.apache.org/jira/browse/BEAM-101
>             Project: Beam
>          Issue Type: New Feature
>          Components: beam-model
>            Reporter: Robert Bradshaw
>
> For some applications, it's useful to declare a pane/window to be emitted (or finished)
based on its contents. The simplest of these is the AfterCount trigger, but more sophisticated
predicates could be constructed.
> The requirements for consistent trigger firing are essentially that the state of the
trigger form a lattice and that the "should fire?" question is a monotonic predicate on the
lattice. Basically it asks "are we high enough up the lattice?"
> Because the element types may change between the application of Windowing and the actuation
of the trigger, one idea is to extract the relevant data from the element at Windowing and
pass it along implicitly where it can be combined and inspected in a type safe way later (similar
to how timestamps and windows are implicitly passed with elements).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message