flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-6014) Allow the registration of state objects in checkpoints
Date Mon, 20 Mar 2017 11:03:41 GMT

    [ https://issues.apache.org/jira/browse/FLINK-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15932493#comment-15932493

ASF GitHub Bot commented on FLINK-6014:

Github user StephanEwen commented on the issue:

    Can you give me some background why you want to make also `PendingCheckpoint` register
its state immediately (and not only upon completion)?
    I see no problem with that, just want to double check whether we are changing the assumption
from the original design doc, where you suggested that shared state can only be referenced
by another checkpoint, if it is already part of a committed checkpoint.

> Allow the registration of state objects in checkpoints
> ------------------------------------------------------
>                 Key: FLINK-6014
>                 URL: https://issues.apache.org/jira/browse/FLINK-6014
>             Project: Flink
>          Issue Type: Sub-task
>          Components: State Backends, Checkpointing
>            Reporter: Xiaogang Shi
>            Assignee: Xiaogang Shi
> This issue is the very first step towards incremental checkpointing. We introduce a new
state handle named {{CompositeStateHandle}} to be the base of the snapshots taken by task
components.  Known implementation may include {{KeyedStateHandle}} (for {{KeyedStateBackend}}s),
{{SubtaskState}} (for subtasks, splits of {{JobVertex}}) and {{TaskState}} (for {{JobVertex}}s).
> Each {{CompositeStateHandle}} is composed of a collection of {{StateObject}s. It should
register all its state objects in {{StateRegistry}} when its checkpoint is added into {{CompletedCheckpointStore}}
(i.e., a pending checkpoint completes or a complete checkpoint is reloaded in the recovery).

> When a completed checkpoint is moved out of the {{CompletedCheckpointStore}}, we should
not simply discard all state objects in the checkpoint. With the introduction of incremental
checkpointing, a {{StateObject}} may be referenced by different checkpoints. We should unregister
all the state objects contained in the {{StateRegistry}} first. Only those state objects that
are not referenced by any checkpoint can be deleted.

This message was sent by Atlassian JIRA

View raw message