flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vahid Hashemian (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7883) Make savepoints atomic with respect to state and side effects
Date Sat, 18 May 2019 15:46:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16843189#comment-16843189

Vahid Hashemian commented on FLINK-7883:

We are also very interested in this enhancement that lets bounded consumption by {{FlinkKafkaConsumer}}.

> Make savepoints atomic with respect to state and side effects
> -------------------------------------------------------------
>                 Key: FLINK-7883
>                 URL: https://issues.apache.org/jira/browse/FLINK-7883
>             Project: Flink
>          Issue Type: Improvement
>          Components: API / DataStream, Connectors / Kafka, Runtime / State Backends
>    Affects Versions: 1.3.2, 1.4.0
>            Reporter: Antoine Philippot
>            Priority: Major
> For a cancel with savepoint command, the JobManager trigger the cancel call once the
savepoint is finished, but during the savepoint execution, kafka source continue to poll new
messages which will not be part of the savepoint and will be replayed on the next application
> A solution could be to stop fetching the source stream task before triggering the savepoint.
> I suggest to add an interface {{StoppableFetchingSourceFunction}} with a method {{stopFetching}}
that existant SourceFunction implementations could implement.
> We can add a {{stopFetchingSource}} property in 
>  {{CheckpointOptions}} class to pass the desired behaviour from {{JobManager.handleMessage(CancelJobWithSavepoint)}}
to {{SourceStreamTask.triggerCheckpoint}}

This message was sent by Atlassian JIRA

View raw message