flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7700) State merging in RocksDB backend leaves old state
Date Wed, 27 Sep 2017 11:17:01 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16182394#comment-16182394

ASF GitHub Bot commented on FLINK-7700:

GitHub user aljoscha opened a pull request:


    [FLINK-7700] Fix RocksDB state merging

    ## What is the purpose of the change
    See title
    ## Brief change log
    This fixes state merging in the respective classes and also consolidates the different
state tests into `StateBackendTestBase` (this helped discover the bug in the first place).
    ## Verifying this change
    See newly added tests that also verify that the backend is empty at the end of the test.
    ## Does this pull request potentially affect one of the following parts:
      - Dependencies (does it add or upgrade a dependency): no
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
      - The serializers: no
      - The runtime per-record code paths (performance sensitive): yes, when merging windows
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing,
Yarn/Mesos, ZooKeeper: no
    ## Documentation
      - Does this pull request introduce a new feature? no
    R: @StefanRRichter could you please review this?

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/aljoscha/flink consolidate-state-tests

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4731
commit 910848fe2f7ada07283414adc4cbd7d86c3d1c2c
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2016-11-04T09:17:55Z

    [FLINK-5619] Add numStateEntries() method for all keyed backends
    This also adds a test for this in StateBackendTestBase

commit 6a29449b5a5bf71dc835d6640890a42951b7b75c
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-09-27T09:37:09Z

    [FLINK-7700] Fix RocksDB ListState merging
    Before, the merged state was not cleared.

commit 1e80aeda0660249dfab9f2131781906ab3c23cbe
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-09-27T09:38:13Z

    [FLINK-5619] Consolidate ListState Tests in StateBackendTestBase

commit a1b2d87218df739ecd3fed3981c59b5cc207fc5f
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-09-27T10:53:08Z

    [FLINK-7700] Fix RocksDB ReducingState merging
    Before, the merged state was not cleared.

commit b0715950d18f7139802597b2a9b0b2b8ec728910
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-09-27T10:54:19Z

    [FLINK-5619] Consolidate ReducingState Tests in StateBackendTestBase

commit f82994af15620bdb5239d44c721aa20b95771639
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-09-27T11:01:55Z

    [FLINK-7700] Fix RocksDB ReducingState merging
    Before, the merged state was not cleared.

commit f9c361ef16538b60446af9dc010d5cf4d5ab8fea
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-09-27T11:02:57Z

    [FLINK-5619] Consolidate AggregatingState Tests in StateBackendTestBase


> State merging in RocksDB backend leaves old state
> -------------------------------------------------
>                 Key: FLINK-7700
>                 URL: https://issues.apache.org/jira/browse/FLINK-7700
>             Project: Flink
>          Issue Type: Bug
>          Components: State Backends, Checkpointing
>    Affects Versions: 1.4.0, 1.3.2
>            Reporter: Aljoscha Krettek
>            Assignee: Aljoscha Krettek
>            Priority: Blocker
>             Fix For: 1.4.0, 1.3.3
> {{RocksDBAggregatingState.mergeNamespaces(...)}}, {{RocksDBReducingState.mergeNamespaces(...)}},
and {{RocksDBListState.mergeNamespaces(...)}} don't remove state from the old location when
merging. This means that merged state will never be cleaned up and will accumulate.

This message was sent by Atlassian JIRA

View raw message