flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aljoscha <...@git.apache.org>
Subject [GitHub] flink pull request #2572: [FLINK-4552] Refactor WindowOperator/Trigger Tests
Date Thu, 29 Sep 2016 15:51:52 GMT
GitHub user aljoscha opened a pull request:

    https://github.com/apache/flink/pull/2572

    [FLINK-4552] Refactor WindowOperator/Trigger Tests

    This builds on #2570 
    
    Before, tests for WindowOperator, WindowAssigner, Trigger and WindowFunction
    were all conflated in WindowOperatorTest. All of these tested that a
    certain combination of a Trigger, WindowAssigner and WindowFunction produce
    the expected output.
    
    This change modularizes these tests and spreads them out across multiple
    files. For example, one per trigger/window assigner.
    
    The new WindowOperatorTest tests verify that the interaction between
    WindowOperator and the various other parts works as expected, that the
    correct methods on Trigger and WindowFunction are called at the expected
    time and that snapshotting, timers, cleanup etc. work correctly. These tests
    also verify that the different state types and WindowFunctions work correctly.
    
    For trigger tests this introduces TriggerTestHarness. This can be used
    to inject elements into Triggers they fire at the correct times. The
    actual output of the WindowFunction is not important for these tests.
    The new tests also make sure that triggers correctly clean up state and timers.
    
    WindowAssigner tests verify the behaviour of window assigners in isolation.
    They also test, for example, whether offset parameter of time-based windows
    work correctly.
    
    R: @StephanEwen @StefanRRichter @kl0u for review

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/aljoscha/flink window-test-refactor

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/2572.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2572
    
----
commit 1a09d9032bf5683a378a7fc8dc480f2d14c5924d
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2016-09-25T18:58:16Z

    Rename TimeServiceProvider to ProcessingTimeService
    
    The name is clashing with the soon-to-be-added
    TimerService/InternalTimerService which is meant as an interface for
    dealing with both processing time and event time.
    
    TimeServiceProvided is renamed to ProcessingTimeService to reflect the
    fact that it is a low-level utility that only deals with "physical"
    processing-time trigger tasks.

commit 758827c3c8508ef9ef2ec079ff3a8469d0096ca8
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2016-09-28T13:10:35Z

    Use OperatorTestHarness and TestProcessingTimeService in Kafka Tests

commit f6dd9c74dc2c58c4263fb6d084651b514898d47a
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2016-09-28T14:35:33Z

    Use Processing-Time Service of TestHarness in WindowOperatorTest
    
    Before, this was manually creating a TestProcessingTimeService, now,
    we're using the one that is there by default in
    OneInputStreamOperatorTestHarness.

commit 65389d66c5586e6707b7a6bf48df512354fac085
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2016-09-28T14:43:40Z

    Refactor OperatorTestHarness to always use TestProcessingTimeService
    
    Before, this would allow handing in a custom ProcessingTimeService but
    this was in reality always TestProcessingTimeService.

commit 1d013bcacc040552e5783c64d094ec309014457b
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2016-09-28T13:12:26Z

    Use TestHarness Processing-time Facility in BucketingSinkTest
    
    Before, this was manually creating a TestProcessingTimeService. Now we
    use the one that is there by default in
    OneInputStreamOperatorTestHarness.

commit eaf3dd00fefeb2487c7cafff6337123cbe42874b
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2016-09-28T13:32:24Z

    Use OperatorTestHarness in AlignedWindowOperator Tests

commit b597d2ef50c27554b83fddaff8873107265340d4
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2016-09-29T14:04:29Z

    Refactor Operator TestHarnesses to use Common Base Class
    
    This also introduces KeyedTwoInputStreamOperatorTestHarness which
    is similar to KeyedOneInputStreamOperatorTestHarness

commit 58b16b26e07b6100f89e9deec63f0decb751f0e6
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2016-09-26T14:21:51Z

    [FLINK-3674] Add an interface for Time aware User Functions
    
    This moves the event-time/processing-time trigger code from
    WindowOperator behind a well defined interface that can be used by
    operators (and user functions).
    
    InternalTimerService is the new interface that has the same
    functionality that WindowOperator used to have. TimerService is the user
    facing interface that does not allow dealing with namespaces/payloads
    and also does not allow deleting timers. There is a default
    implementation in HeapInternalTimerService that can checkpoint timers to
    a stream and also restore from a stream. Right now, this is managed in
    AbstractStreamOperator and operators can ask for an
    InternalTimerService.
    
    This also adds tests for HeapInternalTimerService.
    
    This adds two new user functions:
     - TimelyFlatMapFunction: an extension of FlatMapFunction that also
       allows querying time and setting timers
     - TimelyCoFlatMapFunction: the same, but for CoFlatMapFunction
    
    There are two new StreamOperator implementations for these that use the
    InternalTimerService interface.
    
    This also adds tests for the two new operators.
    
    This also adds the new interface KeyContext that is used for
    setting/querying the current key context for state and timers. Timers
    are always scoped to a key, for now.
    
    Also, this moves the handling of watermarks for both one-input and
    two-input operators to AbstractStreamOperators so that we have a central
    ground-truth.

commit e351b4a409e50c53645ebf7bdbec263148fa956b
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2016-09-05T10:01:11Z

    [FLINK-4552] Refactor WindowOperator/Trigger Tests
    
    Before, tests for WindowOperator, WindowAssigner, Trigger and WindowFunction
    were all conflated in WindowOperatorTest. All of these tested that a
    certain combination of a Trigger, WindowAssigner and WindowFunction produce
    the expected output.
    
    This change modularizes these tests and spreads them out across multiple
    files. For example, one per trigger/window assigner.
    
    The new WindowOperatorTest tests verify that the interaction between
    WindowOperator and the various other parts works as expected, that the
    correct methods on Trigger and WindowFunction are called at the expected
    time and that snapshotting, timers, cleanup etc. work correctly. These tests
    also verify that the different state types and WindowFunctions work correctly.
    
    For trigger tests this introduces TriggerTestHarness. This can be used
    to inject elements into Triggers they fire at the correct times. The
    actual output of the WindowFunction is not important for these tests.
    The new tests also make sure that triggers correctly clean up state and timers.
    
    WindowAssigner tests verify the behaviour of window assigners in isolation.
    They also test, for example, whether offset parameter of time-based windows
    work correctly.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message