flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chengzhi Zhao <w.zhaocheng...@gmail.com>
Subject Fwd: Question about flink checkpoint
Date Tue, 06 Feb 2018 18:15:38 GMT
Hey, I am new to flink and I have a question and want to see if anyone can
help here.

So we have a s3 path that flink is monitoring that path to see new files

val avroInputStream_activity = env.readFile(format, path,
FileProcessingMode.PROCESS_CONTINUOUSLY, 10000)

I am doing both internal and external check pointing and let's say there is
a bad file came to the path and flink will do several retries. I want to
take those bad files and let the process continue. However, since the file
path persist in the checkpoint, when I try to resume from external
checkpoint, it threw the following error on no file been found.

java.io.IOException: Error opening the Input Split s3a://myfile [0,904]: No
such file or directory: s3a://myfile

Is there a way to skip this bad file and move on?
Thanks in advance.

Chengzhi Zhao

View raw message