flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-4455) Replace ActorGateways in NetworkEnvironment by interfaces
Date Thu, 01 Sep 2016 13:44:20 GMT

    [ https://issues.apache.org/jira/browse/FLINK-4455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15455431#comment-15455431

ASF GitHub Bot commented on FLINK-4455:

Github user StephanEwen commented on the issue:

    +1 for moving the `PartitionStateChecker` and `ResultPartitionConsumableNotifier` out
of the `NetworkEnvironment`.
    Few questions and comments:
      - Do we need an extra ExecutorService in the TaskManager? I have been digging through
a bunch of thread dumps over time and there are already many threads and pools already. I
would really like to avoid having yet another Thread pool (creating thread pools should be
an extremely careful decision).
      The Akka thread pool executor is quite over-provisioned for the few actors we actually
use. I think it is perfectly feasible to use that one for the few extra futures introduced
here. In any case, if not reusing the Akka executor pool, then the thread pool needs to be
shut down in the TaskManager runner. Otherwise it creates a leak when running successive local
Flink jobs.
      - I am a bit consumed about the `SlotEnvironment`. Maybe it is mainly the name, but
what does it have to do with the slots? Is it not more like a network-messages specific *JobManager
      - The `ResultPartitionConsumableNotifier` could be per `Task` - that way, future multi-JobManager
assiciations would work seamlessly and it could directly call `fail(...)` on the Task without
having to go through the `TaskManager`. It could probably leave the TaskManager out of the
picture completely.

> Replace ActorGateways in NetworkEnvironment by interfaces
> ---------------------------------------------------------
>                 Key: FLINK-4455
>                 URL: https://issues.apache.org/jira/browse/FLINK-4455
>             Project: Flink
>          Issue Type: Improvement
>          Components: Network, TaskManager
>            Reporter: Till Rohrmann
>            Assignee: Till Rohrmann
> The {{NetworkEnvironment}} communicates with the outside world ({{TaskManager}} and {{JobManager}})
via {{ActorGateways}}. This bakes in the dependency on actors. 
> In terms of modularization and an improved abstraction (especially wrt Flip-6) I propose
to replace the {{ActorGateways}} by interfaces which exposes the required methods. The current
implementation would then simply wrap the method calls in messages and send them via the {{ActorGateway}}
to the recipient.
> In Flip-6 the {{JobMaster}} and the {{TaskExecutor}} could simply implement these interfaces
as part of their RPC contract.

This message was sent by Atlassian JIRA

View raw message