flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tillrohrmann <...@git.apache.org>
Subject [GitHub] flink pull request #4728: [FLINK-7668] Add ExecutionGraphCache for Execution...
Date Tue, 26 Sep 2017 14:59:19 GMT
GitHub user tillrohrmann opened a pull request:

    https://github.com/apache/flink/pull/4728

    [FLINK-7668] Add ExecutionGraphCache for ExecutionGraph based REST handlers

    ## What is the purpose of the change
    
    The ExecutionGraphCache replaces the ExecutionGraphHolder. Unlike the latter, the former
    does not expect the AccessExecutionGraph to be the true ExecutionGraph. Instead it assumes
    it to be the ArchivedExecutionGraph. Therefore, it invalidates the cache entries after
    a given time to live period. This will trigger requesting the AccessExecutionGraph again
    and, thus, updating the ExecutionGraph information for the ExecutionGraph based REST
    handlers.
    
    In order to avoid memory leaks, the WebRuntimeMonitor starts now a periodic cleanup task
    which triggers ExecutionGraphCache.cleanup. This methods releases all cache entries which
    have exceeded their time to live. Currently it is set to 20 * refreshInterval of the
    web gui.
    
    ## Brief change log
    
    - Rename `ExecutionGraphHolder` into `ExecutionGraphCache`
    - Make `ExecutionGraphCache` thread safe by using a `ConcurrentHashMap`
    - Add `ExecutionGraph` request logic depending on the time to live for each cache entry
    - Add periodic cleanup task to `WebRuntimeMonitor` which triggers `ExecutionGraphCache#cleanup`
    
    ## Verifying this change
    
    This change added tests and can be verified as follows:
    
    - `ExecutionGraphCacheTest` adds multiple tests
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): (no)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
      - The serializers: (no)
      - The runtime per-record code paths (performance sensitive): (no)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing,
Yarn/Mesos, ZooKeeper: (no)
    
    ## Documentation
    
      - Does this pull request introduce a new feature? (no)
      - If yes, how is the feature documented? (not applicable)
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tillrohrmann/flink refreshExecutionGraphHolder

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/4728.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4728
    
----
commit ed2d5482c315d64216e4a645b68e7b48dfb89456
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-09-21T08:53:24Z

    [FLINK-7650] [flip6] Port JobCancellationHandler to new REST endpoint
    
    Let the JobCancellationHandler implement the LegacyRestHandler interface. Moreover,
    this commit adds the DELETE method to HttpMethodWrapper and the
    RestServerEndpoint#registerHandler method.

commit e83065fa8102c10a10cdc981f0c7608577c20c34
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-09-21T14:47:18Z

    [FLINK-7649] [flip6] Extend JobTerminationHandler to support stop
    
    Rename the JobCancellationHandler into JobTerminationHandler which is now responsible
    for terminating jobs. Moreover, this commits adds two termination modes, cancel and stop,
    which are specified by providing a query parameter.

commit 02e345b8f34890d67230d0a75a818a8e3b65cf2b
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-09-22T11:31:12Z

    [FLINK-7667] [flip6] Use ArchivedExecutionGraph as serializable AccessExecutionGraph
    
    This commit removes AccessExecutionGraph#getCheckpointCoordinator and changes the
    AccessExecutionGraph#getJobCheckpointSettings into #getJobCheckpointConfiguration.
    The JobCheckpointConfiguration only contains the CheckpointCoordinator relevant
    configuration settings and excludes the serialized state backend and the
    serialized master hooks. That way we don't send unnecessary information when
    the ArchivedExecutionGraph is requested.

commit ab4b21fdf0c00d531d04de6c6272c2812cb61e51
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-09-25T13:29:59Z

    [FLINK-7668] Add ExecutionGraphCache for ExecutionGraph based REST handlers
    
    The ExecutionGraphCache replaces the ExecutionGraphHolder. Unlike the latter, the former
    does not expect the AccessExecutionGraph to be the true ExecutionGraph. Instead it assumes
    it to be the ArchivedExecutionGraph. Therefore, it invalidates the cache entries after
    a given time to live period. This will trigger requesting the AccessExecutionGraph again
    and, thus, updating the ExecutionGraph information for the ExecutionGraph based REST
    handlers.
    
    In order to avoid memory leaks, the WebRuntimeMonitor starts now a periodic cleanup task
    which triggers ExecutionGraphCache.cleanup. This methods releases all cache entries which
    have exceeded their time to live. Currently it is set to 20 * refreshInterval of the
    web gui.

----


---

Mime
View raw message