flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Biao Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-12574) using sink StreamingFileSink files are overwritten when resuming application causing data loss
Date Fri, 24 May 2019 07:21:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16847298#comment-16847298
] 

Biao Liu commented on FLINK-12574:
----------------------------------

Hi [~yitz589],

In my opinion this is more like a customized issue. How about change a base directory every
time after resuming the job from some checkpoint which is not the latest checkpoint? Maybe
with an incremental id suffix? 

> using sink StreamingFileSink files are overwritten when resuming application causing
data loss
> ----------------------------------------------------------------------------------------------
>
>                 Key: FLINK-12574
>                 URL: https://issues.apache.org/jira/browse/FLINK-12574
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / FileSystem
>    Affects Versions: 1.8.0
>            Reporter: yitzchak lieberman
>            Priority: Critical
>
> when part files are saved to s3 bucket (with bucket assigner) with simple names such
as:
> part-0-0 and part-1-2
> restarting or resuming application causes checkpoint id to start from 0 and old files
will be replaced by new part files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message