flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dachuan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-12069) Add proper lifecycle management for intermediate result partitions
Date Mon, 08 Apr 2019 11:46:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-12069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812362#comment-16812362

dachuan commented on FLINK-12069:

Seems it's similar topic with FLINK-12070 Make blocking result partitions consumable multiple
times, could you please help check whether it's duplicate one with FLINK-12070  or any further
work to be done? [~till.rohrmann]

> Add proper lifecycle management for intermediate result partitions
> ------------------------------------------------------------------
>                 Key: FLINK-12069
>                 URL: https://issues.apache.org/jira/browse/FLINK-12069
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Coordination, Runtime / Network
>    Affects Versions: 1.8.0, 1.9.0
>            Reporter: Till Rohrmann
>            Priority: Critical
> In order to properly execute batch jobs, we should make the lifecycle management of intermediate
result partitions the responsibility of the {{JobMaster}}/{{Scheduler}} component. The {{Scheduler}}
knows best when an intermediate result partition is no longer needed and, thus, can be freed.
So for example, a blocking intermediate result should only be released after all subsequent
blocking intermediate results have been completed in order to speed up potential failovers.
> Moreover, having explicit control over intermediate result partitions, could also enable
use cases like result partition sharing between jobs and even across clusters (by simply not
releasing the result partitions). 

This message was sent by Atlassian JIRA

View raw message