flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-9938) State TTL cleanup during full state scan upon checkpointing
Date Fri, 03 Aug 2018 14:37:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-9938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568263#comment-16568263
] 

ASF GitHub Bot commented on FLINK-9938:
---------------------------------------

azagrebin commented on a change in pull request #6460: [FLINK-9938] Clean up full snapshot
from expired state with TTL
URL: https://github.com/apache/flink/pull/6460#discussion_r207565319
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/ttl/TtlStateFactory.java
 ##########
 @@ -176,10 +179,23 @@ private TtlStateFactory(KeyedStateFactory originalStateFactory, StateTtlConfigur
 			new TtlFoldFunction<>(foldingStateDescriptor.getFoldFunction(), ttlConfig, timeProvider,
initAcc),
 			new TtlSerializer<>(stateDesc.getSerializer()));
 		return (IS) new TtlFoldingState<>(
-			originalStateFactory.createInternalState(namespaceSerializer, ttlDescriptor),
+			originalStateFactory.createInternalState(namespaceSerializer, ttlDescriptor, getSnapshotFilter()),
 			ttlConfig, timeProvider, stateDesc.getSerializer());
 	}
 
+	@SuppressWarnings("unchecked")
+	private <TTLV> StateSnapshotFilter<TTLV> getSnapshotFilter() {
+		if (!ttlConfig.getCleanupStrategies().inFullSnapshot()) {
+			return StateSnapshotFilter.snapshotAll();
+		} else if (stateDesc instanceof ListStateDescriptor) {
 
 Review comment:
   I think, it is better to keep state descriptors only for user config.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> State TTL cleanup during full state scan upon checkpointing
> -----------------------------------------------------------
>
>                 Key: FLINK-9938
>                 URL: https://issues.apache.org/jira/browse/FLINK-9938
>             Project: Flink
>          Issue Type: Improvement
>          Components: State Backends, Checkpointing
>    Affects Versions: 1.6.0
>            Reporter: Andrey Zagrebin
>            Assignee: Andrey Zagrebin
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.6.0
>
>
> We can try to piggyback full state scan during certain checkpoint processes in backends,
check TTL expiration for every entry and evict expired to speed up cleanup.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message