flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Biao Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-12172) Flink fails to close pending BucketingSink
Date Tue, 16 Apr 2019 09:26:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-12172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818821#comment-16818821
] 

Biao Liu commented on FLINK-12172:
----------------------------------

Hello,

AFAIK, there is no built-in source that handles automatically .valid-length files. If your
file system supports "truncate" semantic, that would be much easier. All you need to do is
filtering out the files start with ".". "StreamExecutionEnvironment.readTextFile" could satisfy
you since it automatically adds a file filter to do this.

For the window part of your question. Sorry I didn't get your point.

BTW, I think the properest place to discuss these question is user mailing list. I don't think
there is any bug in BuckingSink in your case.

> Flink fails to close pending BucketingSink
> ------------------------------------------
>
>                 Key: FLINK-12172
>                 URL: https://issues.apache.org/jira/browse/FLINK-12172
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / FileSystem
>    Affects Versions: 1.7.2
>            Reporter:  Mario Georgiev
>            Priority: Major
>
> Hello,
> The problem is if you have a BucketingSink, the following case may occur :
> Let's say you have a 2019-04-12–12 bucket created with several files inside which are
pending/finished
>  You create a savepoint and shut down the job
>  After an hour for instance you start the job from the savepoint and a new bucket is
created, 2019-04-16 for instance. 
>  The problem is that the .pending ones from the old buckets seem to never be moved to
finished state if there is a new hourly bucket created.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message