flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-6014) Allow the registration of state objects in checkpoints
Date Tue, 21 Mar 2017 01:43:41 GMT

    [ https://issues.apache.org/jira/browse/FLINK-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15933959#comment-15933959

ASF GitHub Bot commented on FLINK-6014:

Github user shixiaogang commented on the issue:

    Hi @StephanEwen The main reason is that we must have methods to delete those unshared
objects in failed `PendingCheckpoint`s.  The `discardState()` method is called when either
the `PendingCheckpoint` fails or the `CompletedCheckpoint` is subsumed. Under current settings,
the `discardState()` is supposed to delete only those unshared objects, and the shared objects
are deleted by the `StateRegistry`.  Hence, we must register those state handles once they
are received  so that their shared objects can be correctly deleted.

> Allow the registration of state objects in checkpoints
> ------------------------------------------------------
>                 Key: FLINK-6014
>                 URL: https://issues.apache.org/jira/browse/FLINK-6014
>             Project: Flink
>          Issue Type: Sub-task
>          Components: State Backends, Checkpointing
>            Reporter: Xiaogang Shi
>            Assignee: Xiaogang Shi
> This issue is the very first step towards incremental checkpointing. We introduce a new
state handle named {{CompositeStateHandle}} to be the base of the snapshots taken by task
components.  Known implementation may include {{KeyedStateHandle}} (for {{KeyedStateBackend}}s),
{{SubtaskState}} (for subtasks, splits of {{JobVertex}}) and {{TaskState}} (for {{JobVertex}}s).
> Each {{CompositeStateHandle}} is composed of a collection of {{StateObject}s. It should
register all its state objects in {{StateRegistry}} when its checkpoint is added into {{CompletedCheckpointStore}}
(i.e., a pending checkpoint completes or a complete checkpoint is reloaded in the recovery).

> When a completed checkpoint is moved out of the {{CompletedCheckpointStore}}, we should
not simply discard all state objects in the checkpoint. With the introduction of incremental
checkpointing, a {{StateObject}} may be referenced by different checkpoints. We should unregister
all the state objects contained in the {{StateRegistry}} first. Only those state objects that
are not referenced by any checkpoint can be deleted.

This message was sent by Atlassian JIRA

View raw message