flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yun Tang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-11874) Split CheckpointStorage interface to distinguish JM and TM side
Date Tue, 12 Mar 2019 04:31:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-11874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16790221#comment-16790221
] 

Yun Tang commented on FLINK-11874:
----------------------------------

[~yanghua] Thanks for your suggestion.

I thought I had given the why-what-how in the JIRA's description:
*why*: Current interface {{CheckpointStorage}} mixed JM and TM side, which would be confusing
for users and developers to distinguish them. Especially when developers implement some function
which should not be called in TM side such as init directories.
*what*: {{CheckpointStorage}} would not exist in TM side but also in JM side, in which they
acted as different roles.
*how*: Split JM's side methods out of current {{CheckpointStorage}} interface to a new interface
such as {{CheckpointCoordinatorStorage}}. By using this, we could cast {{CheckpointStorage}}
to {{CheckpointCoordinatorStorage}} within checkpoint coordinator so that those methods could
be called only in JM side.

This JIRA was first inspired when we review [FLINK-11696|https://github.com/apache/flink/pull/7942#discussion_r264335302]'s
PR, it would be really very helpful to point out anything that should do. 

If you found anything deserved to be discussed more clearly to send a design proposal, you
could leave your suggestions and concerns in details :)

> Split CheckpointStorage interface to distinguish JM and TM side
> ---------------------------------------------------------------
>
>                 Key: FLINK-11874
>                 URL: https://issues.apache.org/jira/browse/FLINK-11874
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Checkpointing
>            Reporter: Yun Tang
>            Assignee: Yun Tang
>            Priority: Major
>             Fix For: 1.9.0
>
>
> Currently, interface {{CheckpointStorage}} mixed JM and TM side, which would be confusing
for users and developers to distinguish them. 
> Take [FLINK-11696|https://issues.apache.org/jira/browse/FLINK-11696] as an example, the
directories should only be created once from JM side. However, since we mixed the JM and TM
side, TMs would also create directories again.
> We could let interface {{CheckpointStorage}} only has two methods:
> {code:java}
> CheckpointStreamFactory resolveCheckpointStorageLocation(
> 			long checkpointId,
> 			CheckpointStorageLocationReference reference)
> CheckpointStateOutputStream createTaskOwnedStateStream()
> {code}
> And a new interface {{ChekcpointCoordinatorStorgae}} could be introduced extending {{CheckpointStorage}}
and have methods below:
> {code:java}
> boolean supportsHighlyAvailableStorage()
> boolean hasDefaultSavepointLocation()
> CompletedCheckpointStorageLocation resolveCheckpoint(String externalPointer)
> CheckpointStorageLocation initializeLocationForCheckpoint(long checkpointId)
> CheckpointStorageLocation initializeLocationForSavepoint(
> 			long checkpointId,
> 			@Nullable String externalLocationPointer)
> {code}
> With this refactor, JM would only use cast {{ChekcpointCoordinatorStorgae}} so that we
could shield TMs from accidentally calling unexpected methods.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message