flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From StephanEwen <...@git.apache.org>
Subject [GitHub] flink pull request: [FLINK-3051] Control the maximum number of con...
Date Wed, 25 Nov 2015 19:51:14 GMT
GitHub user StephanEwen opened a pull request:


    [FLINK-3051] Control the maximum number of concurrent checkpoints

    This change introduces a parameter that lets users control at most how many checkpoints
    should be in progress at any given point in time. This is very useful for
    cases where the best checkpoint interval is not easy to determine, or where once in
    a while checkpoints may take long. Previously, such situations lead to checkpoint
    queue-up, where the system ended up being more busy with checkpoints than with processing.
    After this change, the checkpoint rate will by default sort of self-regulate: The checkpoint
interval is the most frequent time at which checkpoints will occur, but they will occur slower
if they take longer than that interval.
    ### Checkpoint Config
    This also introduces the CheckpointConfig class that holds all checkpointing related parameters
to more simply pass them between environment, streamgraph, etc...
    ### Default number of concurrent checkpoints
    Previously, the maximum number of concurrent checkpoints was implicitly infinite.
    I would suspect that most people will want to run the system such that only one checkpoint
is ever concurrently active, so I set the default value for this to 1.
    In some corner cases, it may be interesting to set that value higher.
    ### WIP for delay between checkpoints
    This also contains some WIP to define a minimum time between checkpoint attempts. That
flag would tell the system not only to not do more than one checkpoint concurrently, but to
leave at least a certain time between completion of one checkpoint, and the triggering of
the next. It basically defines a guaranteed time that is "work only" between checkpoints.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/StephanEwen/incubator-flink checkpoints_configure

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1408
commit e4d063867ac09cbec6666ecbca92816cfb6faf6c
Author: Stephan Ewen <sewen@apache.org>
Date:   2015-11-19T17:03:55Z

    [hotfix] Java-7-ify the ExecutionConfig class

commit 79def44a2e973c8b6817821d5d7342731f7e7aa2
Author: Stephan Ewen <sewen@apache.org>
Date:   2015-11-19T18:05:47Z

    [FLINK-3051] [streaming] Add mechanisms to control the maximum number of concurrent checkpoints


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message