flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chesnay Schepler (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (FLINK-12069) Add proper lifecycle management for intermediate result partitions
Date Fri, 24 May 2019 07:40:00 GMT

     [ https://issues.apache.org/jira/browse/FLINK-12069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chesnay Schepler reassigned FLINK-12069:

    Assignee: Chesnay Schepler

> Add proper lifecycle management for intermediate result partitions
> ------------------------------------------------------------------
>                 Key: FLINK-12069
>                 URL: https://issues.apache.org/jira/browse/FLINK-12069
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Coordination, Runtime / Network
>    Affects Versions: 1.8.0, 1.9.0
>            Reporter: Till Rohrmann
>            Assignee: Chesnay Schepler
>            Priority: Critical
> In order to properly execute batch jobs, we should make the lifecycle management of intermediate
result partitions the responsibility of the {{JobMaster}}/{{Scheduler}} component. The {{Scheduler}}
knows best when an intermediate result partition is no longer needed and, thus, can be freed.
So for example, a blocking intermediate result should only be released after all subsequent
blocking intermediate results have been completed in order to speed up potential failovers.
> Moreover, having explicit control over intermediate result partitions, could also enable
use cases like result partition sharing between jobs and even across clusters (by simply not
releasing the result partitions). 

This message was sent by Atlassian JIRA

View raw message