flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Till Rohrmann (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-12069) Add proper lifecycle management for intermediate result partitions
Date Tue, 09 Apr 2019 09:52:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-12069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16813196#comment-16813196

Till Rohrmann commented on FLINK-12069:

Hi [~dachuan.qdc], it's true that FLINK-12070 is closely related to this issue. The difference
is that this issue covers the lifecycle management part (communication between the {{JobMaster}}
and the {{TaskExecutor}}) whereas FLINK-12070 is intended to make the blocking {{ResultPartition}}
consumable multiple times (this means that it does not give up memory buffers or writes the
result to a file). Both issues together will allow us to recover from blocking results (the
result itself does not release data when being consumed + the result partition is not being
released once the consumers finishes consumption). One could maybe address both issues together,

> Add proper lifecycle management for intermediate result partitions
> ------------------------------------------------------------------
>                 Key: FLINK-12069
>                 URL: https://issues.apache.org/jira/browse/FLINK-12069
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Coordination, Runtime / Network
>    Affects Versions: 1.8.0, 1.9.0
>            Reporter: Till Rohrmann
>            Priority: Critical
> In order to properly execute batch jobs, we should make the lifecycle management of intermediate
result partitions the responsibility of the {{JobMaster}}/{{Scheduler}} component. The {{Scheduler}}
knows best when an intermediate result partition is no longer needed and, thus, can be freed.
So for example, a blocking intermediate result should only be released after all subsequent
blocking intermediate results have been completed in order to speed up potential failovers.
> Moreover, having explicit control over intermediate result partitions, could also enable
use cases like result partition sharing between jobs and even across clusters (by simply not
releasing the result partitions). 

This message was sent by Atlassian JIRA

View raw message