flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gyfora <...@git.apache.org>
Subject [GitHub] flink pull request: Pluggable state backend added
Date Thu, 14 May 2015 19:34:06 GMT
GitHub user gyfora opened a pull request:


    Pluggable state backend added

    This PR introduces the pluggable state backends using StateHandleProviders and extends
the StateHandle interface with a discard method for cleaning up the unnecessary checkpoints.
    It also adds a statehandle/provider implementation for storing checkpoints in any flink
supported file system such as HDFS or Tachyon.
    The checkpoint coordinator has been modified to properly discard user state handles using
the following logic:
     - If a pending checkpoint expires (by the timer thread) it discards the user state
     - When a successful checkpoint expires (by acquiring following successful ones) it discards
user states
     - When the checkpoint coordinator is shut down it discards all pending and successful
    Travis error:
    I modified the recovery IT case to use the local file system for storing the checkpoints.
Afterwards it checks whether the directory is empty. The test passes all the time ran locally,
but it seems to fail on travis for no apparent reason. Usually a couple of files (2-5) remain
in the checkpoint directory, meaning that almost all of them had been deleted but those. Also
the checkpointing and recovery logic runs fine without test failure.
    I would appreciate some help figuring this out somehow, or trying to reproduce it locally.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/gyfora/flink statehandle

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #676
commit 22b5996e046cf83612fac2cb5aa02f2fd76a7e7b
Author: Gyula Fora <gyfora@apache.org>
Date:   2015-05-07T12:29:23Z

    [streaming] ByteStream and File state handle added

commit fed66675e2e824eee00b88197c5a73882415c919
Author: Gyula Fora <gyfora@apache.org>
Date:   2015-05-14T09:49:30Z

    [streaming] Discard method added to state handle

commit 36474aafe74be9b61a89b5240bbc39f47226da77
Author: Gyula Fora <gyfora@apache.org>
Date:   2015-05-14T19:23:42Z

    [streaming] StateHandleProvider added for configurable state backend


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message