flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2329) Refactor RPCs from within the ExecutionGraph
Date Wed, 08 Jul 2015 15:12:04 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14618751#comment-14618751

ASF GitHub Bot commented on FLINK-2329:

GitHub user tillrohrmann opened a pull request:


    [FLINK-2329] [runtime] Introduces InstanceGateway for RPCs from Executions

    This PR replaces the direct `ActorRef` interaction from within `Executions` with an `InstanceGateway`.
The `InstanceGateway` is an abstraction which encapsulates the RPC logic. This allows to more
easily separate the `Execution` logic from the used Akka framework. 
    The default implementation `AkkaInstanceGateway` simply forwards the messages to the provided
`ActorRef` which represents the remote instance.
    The `InstanceGateway` will allow us to add leader session IDs to all messages, which are
sent to a `TaskManager`, transparently, without having to pass this information through the
`ExecutionGraph` to the `Executions`.
    The additional abstraction layer allows to write more light-weight tests, too. As a consequence,
many test cases which needed an `ActorSystem` to start a simple testing actor which mimicked
the remote end, could now be simplified using a simple `InstanceGateway` implementation and
thus getting rid off testing `ActorSystems`.
    In the wake of refactoring, I also replaced the `AkkaUtils.globalExecutionContext` with
a non global one. At the moment, the `JobManager` and the `TaskManager` both start a single
`ForkJoinPool` which is used as the `ExecutionContext` for all futures and future handlers.
The default parallelism is the number of cores available on the system. It would now also
be possible to use even more fine grained `ExecutionContexts`, e.g. giving each `ExecutionGraph`
a distinct `ExecutionContext`.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tillrohrmann/flink leaderSessionID

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #893
commit 43e15aa73dde20166cd9973cd5418fe352ea02d4
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2015-07-07T09:41:44Z

    [FLINK-2329] [runtime] Introduces InstanceGateway as an abstraction to communicate with
the TaskManager. Replaces AkkaUtils.globalExecutionContext with instance dependent ExecutionContext.
    Add comments for newly introduced classes


> Refactor RPCs from within the ExecutionGraph
> --------------------------------------------
>                 Key: FLINK-2329
>                 URL: https://issues.apache.org/jira/browse/FLINK-2329
>             Project: Flink
>          Issue Type: Sub-task
>          Components: JobManager, TaskManager
>            Reporter: Till Rohrmann
>            Assignee: Till Rohrmann
>             Fix For: 0.10
> Currently, we store an {{ActorRef}} of the TaskManager into an {{Instance}} object. This
{{ActorRef}} is used from within {{Executions}} to interact with the {{TaskManager}}. This
is not a nice abstraction since it does not hide implementation details. 
> Since we need to add a leader session ID to messages sent by the {{Executions}} in order
to support high availability, we would need to make the leader session ID available to the
{{Execution}}. A better solution seems to be to replace the direct {{ActorRef}} interaction
with an instance gateway abstraction which encapsulates the communication logic. Having such
an abstraction, it will be easy to decorate messages transparently with a leader session ID.
Therefore, I propose to refactor the current {{Instance}} communication and to introduce an
{{InstanceGateway}} abstraction.

This message was sent by Atlassian JIRA

View raw message