flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From uce <...@git.apache.org>
Subject [GitHub] flink pull request: [FLINK-2805] [blobmanager] Write JARs to file ...
Date Mon, 05 Oct 2015 15:34:57 GMT
GitHub user uce opened a pull request:


    [FLINK-2805] [blobmanager] Write JARs to file state backend for recovery

    This is a follow up to #1153. I've taken two changes from #1153 for convenience. Other
than that, this is independent.
    When running the `BlobServer` in `RecoveryMode#ZOOKEEPER`, this will upload the JARs to
the configured file system backend (e.g. HDFS). **Important**: it introduces a hard dependency
to have a configured file state backend when running the blob server with recovery. This is
in line with #1153.
    This JAR copying only happens on the server side, e.g. the client uploads to the server
and the server uploads it to the state backend. Same when requesting a locally non-existing
blob: the client requests from the server and the server downloads if not available and then
answers the client.
    There are other ways to implement this, but this one was minimally invasive and fully
circumvents any Akka actor threads for downloading/uploading. A more invasive change could
allow to directly interact with the state backend on the client side [task manager] as well.
This would spread the load better across the cluster in case of a DFS and save unnecessary
network transfers.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/uce/flink hdfs_jars-2805

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1227
commit 68a7ac4abab15dc2ef45546de6a94c27129534dc
Author: Ufuk Celebi <uce@apache.org>
Date:   2015-10-05T12:30:46Z

    [FLINK-2805] Apply RecoveryMode and ConfigConstants changes from #1153

commit 00ea6b02f15f00486c9541af08b8197c41dd94f7
Author: Ufuk Celebi <uce@apache.org>
Date:   2015-10-05T08:05:05Z

    [FLINK-2805] [blobmanager] Write JARs to file state backend for recovery


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message