flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-10946) Resuming Externalized Checkpoint (rocks, incremental, scale up) end-to-end test failed on Travis
Date Tue, 04 Dec 2018 15:07:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-10946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16708829#comment-16708829
] 

ASF GitHub Bot commented on FLINK-10946:
----------------------------------------

tillrohrmann closed pull request #7204: [FLINK-10946] Silent checkpoint async failures in
task executor if job is not running
URL: https://github.com/apache/flink/pull/7204
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java
b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java
index f0ea6d41610..6f8918d41ae 100644
--- a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java
+++ b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java
@@ -415,7 +415,9 @@ public final OperatorSnapshotFutures snapshotState(long checkpointId,
long times
 			String snapshotFailMessage = "Could not complete snapshot " + checkpointId + " for operator
" +
 				getOperatorName() + ".";
 
-			LOG.info(snapshotFailMessage, snapshotException);
+			if (!getContainingTask().isCanceled()) {
+				LOG.info(snapshotFailMessage, snapshotException);
+			}
 			throw new Exception(snapshotFailMessage, snapshotException);
 		}
 


 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Resuming Externalized Checkpoint (rocks, incremental, scale up) end-to-end test failed
on Travis
> ------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-10946
>                 URL: https://issues.apache.org/jira/browse/FLINK-10946
>             Project: Flink
>          Issue Type: Bug
>          Components: E2E Tests
>    Affects Versions: 1.7.0
>            Reporter: Andrey Zagrebin
>            Assignee: Andrey Zagrebin
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.6.3, 1.7.0
>
>
> The test failed 3 times in total during the overall night build, but succeeded 2 times
after restart. It did not fail locally for me.
> Here is a travis build to run it 500 times (reproducable):
> [https://travis-ci.org/azagrebin/flink/builds/457375100]
> {code:java}
> 2018-11-20 11:59:54,673 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Triggering cancellation of task code ArtificalKeyedStateMapper_Avro -> ArtificalOperatorStateMapper
(2/2) (e06b7022f2f2154f2a84206f068ff1fd).
> 2018-11-20 11:59:54,701 INFO  org.apache.flink.streaming.api.operators.AbstractStreamOperator
 - Could not complete snapshot 12 for operator ArtificalKeyedStateMapper_Avro -> ArtificalOperatorStateMapper
(1/2).
> java.io.IOException: Cannot register Closeable, registry is already closed. Closing argument.
> 	at org.apache.flink.util.AbstractCloseableRegistry.registerCloseable(AbstractCloseableRegistry.java:85)
> 	at org.apache.flink.runtime.state.AsyncSnapshotCallable$AsyncSnapshotTask.<init>(AsyncSnapshotCallable.java:123)
> 	at org.apache.flink.runtime.state.AsyncSnapshotCallable$AsyncSnapshotTask.<init>(AsyncSnapshotCallable.java:111)
> 	at org.apache.flink.runtime.state.AsyncSnapshotCallable.toAsyncSnapshotFutureTask(AsyncSnapshotCallable.java:105)
> 	at org.apache.flink.contrib.streaming.state.snapshot.RocksIncrementalSnapshotStrategy.doSnapshot(RocksIncrementalSnapshotStrategy.java:164)
> 	at org.apache.flink.contrib.streaming.state.snapshot.RocksDBSnapshotStrategyBase.snapshot(RocksDBSnapshotStrategyBase.java:128)
> 	at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:496)
> 	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:407)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.checkpointStreamOperator(StreamTask.java:1113)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1055)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:729)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:641)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:586)
> 	at org.apache.flink.streaming.runtime.io.BarrierBuffer.notifyCheckpoint(BarrierBuffer.java:396)
> 	at org.apache.flink.streaming.runtime.io.BarrierBuffer.processBarrier(BarrierBuffer.java:292)
> 	at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:200)
> 	at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:209)
> 	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)
> 	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
> 	at java.lang.Thread.run(Thread.java:748)
> 2018-11-20 11:59:54,702 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Ensuring all FileSystem streams are closed for task Source: Custom Source -> Timestamps/Watermarks
(2/2) (09528d6ab0e1ee87ed21e78139682b18) [CANCELED]
> 2018-11-20 11:59:54,703 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Ensuring all FileSystem streams are closed for task Source: Custom Source -> Timestamps/Watermarks
(1/2) (c98146380e7f559ca18a183f4c0ef12d) [CANCELED]
> 2018-11-20 11:59:54,721 INFO  org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend
 - Deleting existing instance base directory /tmp/flink-io-71851605-f8f2-4d4e-83e7-9b69e0a879ef/job_41e7002f55a128f646117fc14cf858a1_op_StreamMap_7d23c6ceabda05a587f0217e44f21301__2_2__uuid_59f43f20-768f-4117-9a3f-4a101a32b1d2.
> 2018-11-20 11:59:54,724 INFO  org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend
 - Deleting existing instance base directory /tmp/flink-io-71851605-f8f2-4d4e-83e7-9b69e0a879ef/job_41e7002f55a128f646117fc14cf858a1_op_StreamMap_7d23c6ceabda05a587f0217e44f21301__1_2__uuid_f3ce6abc-52dc-4fa1-820e-e015daca418c.
> 2018-11-20 11:59:54,732 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Attempting to cancel task TumblingWindowOperator (1/2) (3ea121867a218e10137c9bfe9ef991b8).
> 2018-11-20 11:59:54,732 INFO  org.apache.flink.runtime.taskmanager.Task             
       - TumblingWindowOperator (1/2) (3ea121867a218e10137c9bfe9ef991b8) switched from RUNNING
to CANCELING.
> 2018-11-20 11:59:54,732 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Triggering cancellation of task code TumblingWindowOperator (1/2) (3ea121867a218e10137c9bfe9ef991b8).
> 2018-11-20 11:59:54,769 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Attempting to cancel task TumblingWindowOperator (2/2) (7a117e9804f71c6a995e31616ee05ec9).
> 2018-11-20 11:59:54,770 INFO  org.apache.flink.runtime.taskmanager.Task             
       - TumblingWindowOperator (2/2) (7a117e9804f71c6a995e31616ee05ec9) switched from RUNNING
to CANCELING.
> 2018-11-20 11:59:54,770 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Triggering cancellation of task code TumblingWindowOperator (2/2) (7a117e9804f71c6a995e31616ee05ec9).
> 2018-11-20 11:59:54,789 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Attempting to cancel task SemanticsCheckMapper -> Sink: Unnamed (1/2) (0a53c5fdd36dffde33ef032b6ffb5307).
> 2018-11-20 11:59:54,789 INFO  org.apache.flink.runtime.taskmanager.Task             
       - SemanticsCheckMapper -> Sink: Unnamed (1/2) (0a53c5fdd36dffde33ef032b6ffb5307)
switched from RUNNING to CANCELING.
> 2018-11-20 11:59:54,789 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Triggering cancellation of task code SemanticsCheckMapper -> Sink: Unnamed (1/2)
(0a53c5fdd36dffde33ef032b6ffb5307).
> 2018-11-20 11:59:54,813 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Attempting to cancel task SemanticsCheckMapper -> Sink: Unnamed (2/2) (a9b08ced9d0d0ed0fb130336340b1a7a).
> 2018-11-20 11:59:54,813 INFO  org.apache.flink.runtime.taskmanager.Task             
       - SemanticsCheckMapper -> Sink: Unnamed (2/2) (a9b08ced9d0d0ed0fb130336340b1a7a)
switched from RUNNING to CANCELING.
> 2018-11-20 11:59:54,813 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Triggering cancellation of task code SemanticsCheckMapper -> Sink: Unnamed (2/2)
(a9b08ced9d0d0ed0fb130336340b1a7a).
> 2018-11-20 11:59:54,824 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Attempting to cancel task SlidingWindowOperator (1/2) (0f6b13316bcfe13e46972d4dd0bc5939).
> 2018-11-20 11:59:54,824 INFO  org.apache.flink.runtime.taskmanager.Task             
       - SlidingWindowOperator (1/2) (0f6b13316bcfe13e46972d4dd0bc5939) switched from RUNNING
to CANCELING.
> 2018-11-20 11:59:54,824 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Triggering cancellation of task code SlidingWindowOperator (1/2) (0f6b13316bcfe13e46972d4dd0bc5939).
> 2018-11-20 11:59:54,831 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Attempting to cancel task SlidingWindowOperator (2/2) (4b0e279fcf1ce9afb5833731a3844319).
> 2018-11-20 11:59:54,831 INFO  org.apache.flink.runtime.taskmanager.Task             
       - SlidingWindowOperator (2/2) (4b0e279fcf1ce9afb5833731a3844319) switched from RUNNING
to CANCELING.
> 2018-11-20 11:59:54,831 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Triggering cancellation of task code SlidingWindowOperator (2/2) (4b0e279fcf1ce9afb5833731a3844319).
> 2018-11-20 11:59:54,844 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Attempting to cancel task SlidingWindowCheckMapper -> Sink: Unnamed (1/2) (f269b77a22ea33b41d58a276154fff75).
> 2018-11-20 11:59:54,844 INFO  org.apache.flink.runtime.taskmanager.Task             
       - SlidingWindowCheckMapper -> Sink: Unnamed (1/2) (f269b77a22ea33b41d58a276154fff75)
switched from RUNNING to CANCELING.
> 2018-11-20 11:59:54,844 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Triggering cancellation of task code SlidingWindowCheckMapper -> Sink: Unnamed
(1/2) (f269b77a22ea33b41d58a276154fff75).
> 2018-11-20 11:59:54,857 INFO  org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend
 - Deleting existing instance base directory /tmp/flink-io-71851605-f8f2-4d4e-83e7-9b69e0a879ef/job_41e7002f55a128f646117fc14cf858a1_op_WindowOperator_0b63e7dd9fb1948bf052174673e64274__1_2__uuid_f01e3e36-565e-4680-b7d8-1f30e6a64474.
> 2018-11-20 11:59:54,742 INFO  org.apache.flink.streaming.api.operators.AbstractStreamOperator
 - Could not complete snapshot 12 for operator ArtificalKeyedStateMapper_Kryo_and_Custom_Stateful
(1/2).
> java.io.IOException: Cannot register Closeable, registry is already closed. Closing argument.
> 	at org.apache.flink.util.AbstractCloseableRegistry.registerCloseable(AbstractCloseableRegistry.java:85)
> 	at org.apache.flink.runtime.state.AsyncSnapshotCallable$AsyncSnapshotTask.<init>(AsyncSnapshotCallable.java:123)
> 	at org.apache.flink.runtime.state.AsyncSnapshotCallable$AsyncSnapshotTask.<init>(AsyncSnapshotCallable.java:111)
> 	at org.apache.flink.runtime.state.AsyncSnapshotCallable.toAsyncSnapshotFutureTask(AsyncSnapshotCallable.java:105)
> 	at org.apache.flink.contrib.streaming.state.snapshot.RocksIncrementalSnapshotStrategy.doSnapshot(RocksIncrementalSnapshotStrategy.java:164)
> 	at org.apache.flink.contrib.streaming.state.snapshot.RocksDBSnapshotStrategyBase.snapshot(RocksDBSnapshotStrategyBase.java:128)
> 	at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:496)
> 	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:407)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.checkpointStreamOperator(StreamTask.java:1113)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1055)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:729)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:641)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:586)
> 	at org.apache.flink.streaming.runtime.io.BarrierBuffer.notifyCheckpoint(BarrierBuffer.java:396)
> 	at org.apache.flink.streaming.runtime.io.BarrierBuffer.processBarrier(BarrierBuffer.java:292)
> 	at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:200)
> 	at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:209)
> 	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)
> 	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
> 	at java.lang.Thread.run(Thread.java:748)
> 2018-11-20 11:59:54,871 INFO  org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend
 - Deleting existing instance base directory /tmp/flink-io-71851605-f8f2-4d4e-83e7-9b69e0a879ef/job_41e7002f55a128f646117fc14cf858a1_op_StreamMap_5271c210329e73bd743f3227edfb3b71__1_2__uuid_0f79c71c-ffe0-4fb6-a16e-a7a7f1c8490d.
> 2018-11-20 11:59:54,747 INFO  org.apache.flink.streaming.api.operators.AbstractStreamOperator
 - Could not complete snapshot 12 for operator ArtificalKeyedStateMapper_Kryo_and_Custom_Stateful
(2/2).
> java.io.IOException: Cannot register Closeable, registry is already closed. Closing argument.
> 	at org.apache.flink.util.AbstractCloseableRegistry.registerCloseable(AbstractCloseableRegistry.java:85)
> 	at org.apache.flink.runtime.state.AsyncSnapshotCallable$AsyncSnapshotTask.<init>(AsyncSnapshotCallable.java:123)
> 	at org.apache.flink.runtime.state.AsyncSnapshotCallable$AsyncSnapshotTask.<init>(AsyncSnapshotCallable.java:111)
> 	at org.apache.flink.runtime.state.AsyncSnapshotCallable.toAsyncSnapshotFutureTask(AsyncSnapshotCallable.java:105)
> 	at org.apache.flink.contrib.streaming.state.snapshot.RocksIncrementalSnapshotStrategy.doSnapshot(RocksIncrementalSnapshotStrategy.java:164)
> 	at org.apache.flink.contrib.streaming.state.snapshot.RocksDBSnapshotStrategyBase.snapshot(RocksDBSnapshotStrategyBase.java:128)
> 	at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:496)
> 	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:407)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.checkpointStreamOperator(StreamTask.java:1113)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1055)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:729)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:641)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:586)
> 	at org.apache.flink.streaming.runtime.io.BarrierBuffer.notifyCheckpoint(BarrierBuffer.java:396)
> 	at org.apache.flink.streaming.runtime.io.BarrierBuffer.processBarrier(BarrierBuffer.java:292)
> 	at org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:200)
> 	at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:209)
> 	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)
> 	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
> 	at java.lang.Thread.run(Thread.java:748)
> 2018-11-20 11:59:54,875 INFO  org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend
 - Deleting existing instance base directory /tmp/flink-io-71851605-f8f2-4d4e-83e7-9b69e0a879ef/job_41e7002f55a128f646117fc14cf858a1_op_StreamMap_5271c210329e73bd743f3227edfb3b71__2_2__uuid_f09e971f-8542-4f0b-9abe-644d702f235f.
> 2018-11-20 11:59:54,873 INFO  org.apache.flink.runtime.taskmanager.Task             
       - ArtificalKeyedStateMapper_Avro -> ArtificalOperatorStateMapper (1/2) (3e5842c071706a82194c67143f6f4a60)
switched from CANCELING to CANCELED.{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message