flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ufuk Celebi (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-1833) Refactor partition availability notification in ExecutionGraph
Date Tue, 07 Apr 2015 09:10:24 GMT
Ufuk Celebi created FLINK-1833:

             Summary: Refactor partition availability notification in ExecutionGraph
                 Key: FLINK-1833
                 URL: https://issues.apache.org/jira/browse/FLINK-1833
             Project: Flink
          Issue Type: Improvement
          Components: Distributed Runtime, JobManager
    Affects Versions: master
            Reporter: Ufuk Celebi

The mechanism to notify the JobManager about available result partitions is hard to understand.
The are two parts to this:

1) JobManager

- The deployment of receivers happens in the Execution class although it is by now totally
unrelated to the state of a specific execution. I propose to move this to the respective IntermediateResultPartition.
- The deployment information for a receiver is spread across different components: when creating
the TaskDeploymentDescriptor and the "caching" of partition infos at the consuming vertex.
This is very hard to follow and results in unnecessary messages being sent (which are discarded
at the TM).

2) TaskManager

- Pipelined results notify where you would expect it in the ResultPartition, but blocking
results don't have an extra message and are implicitly piggy-backed to the final state transition,
after which the job manager deploys receivers if all blocking partitions of a result have
been produced.

This message was sent by Atlassian JIRA

View raw message