beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (BEAM-5372) [Flink Runner] Make minPauseBetweenCheckpoints setting available in FlinkPipelineOptions
Date Fri, 14 Sep 2018 10:16:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-5372?focusedWorklogId=144237&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144237
]

ASF GitHub Bot logged work on BEAM-5372:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 14/Sep/18 10:15
            Start Date: 14/Sep/18 10:15
    Worklog Time Spent: 10m 
      Work Description: aljoscha closed pull request #6378: [BEAM-5372] Make minPauseBetweenCheckpoints
setting available in FlinkPipelineOptions
URL: https://github.com/apache/beam/pull/6378
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
index bf06d0b6699..853d4f39f15 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
@@ -164,6 +164,13 @@ public static StreamExecutionEnvironment createStreamExecutionEnvironment(
                     ? ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION
                     : ExternalizedCheckpointCleanup.DELETE_ON_CANCELLATION);
       }
+
+      long minPauseBetweenCheckpoints = options.getMinPauseBetweenCheckpoints();
+      if (minPauseBetweenCheckpoints != -1) {
+        flinkStreamEnv
+            .getCheckpointConfig()
+            .setMinPauseBetweenCheckpoints(minPauseBetweenCheckpoints);
+      }
     }
 
     applyLatencyTrackingInterval(flinkStreamEnv.getConfig(), options);
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
index 57458658560..9b3c9165208 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
@@ -86,6 +86,12 @@
 
   void setCheckpointTimeoutMillis(Long checkpointTimeoutMillis);
 
+  @Description("The minimal pause before the next checkpoint is triggered.")
+  @Default.Long(-1L)
+  Long getMinPauseBetweenCheckpoints();
+
+  void setMinPauseBetweenCheckpoints(Long minPauseInterval);
+
   @Description(
       "Sets the number of times that failed tasks are re-executed. "
           + "A value of zero effectively disables fault tolerance. A value of -1 indicates
"


 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 144237)
    Time Spent: 1h  (was: 50m)

> [Flink Runner] Make minPauseBetweenCheckpoints setting available in FlinkPipelineOptions
> ----------------------------------------------------------------------------------------
>
>                 Key: BEAM-5372
>                 URL: https://issues.apache.org/jira/browse/BEAM-5372
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-flink
>            Reporter: Daniel Harper
>            Assignee: Daniel Harper
>            Priority: Trivial
>             Fix For: 2.7.0
>
>          Time Spent: 1h
>  Remaining Estimate: 0h
>
> The configuration setting {{minPauseBetweenCheckpoints}} [1] is available in Flink to
allow a grace period when checkpoints runtime is > checkpoint interval. 
> This should be exposed in {{FlinkPipelineOptions}} and {{FlinkExecutionEnvironments}}
to allow users to configure this.
> The default for this value in Flink is 0ms [2] 
> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/api/java/org/apache/flink/streaming/api/environment/CheckpointConfig.html#setMinPauseBetweenCheckpoints-long-
> [2] https://ci.apache.org/projects/flink/flink-docs-release-1.5/api/java/constant-values.html#org.apache.flink.streaming.api.environment.CheckpointConfig.DEFAULT_MIN_PAUSE_BETWEEN_CHECKPOINTS



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message