flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From StephanEwen <...@git.apache.org>
Subject [GitHub] flink issue #2449: [FLINK-4455] [FLINK-4424] [networkenv] Make NetworkEnviro...
Date Thu, 01 Sep 2016 13:43:52 GMT
Github user StephanEwen commented on the issue:

    +1 for moving the `PartitionStateChecker` and `ResultPartitionConsumableNotifier` out
of the `NetworkEnvironment`.
    Few questions and comments:
      - Do we need an extra ExecutorService in the TaskManager? I have been digging through
a bunch of thread dumps over time and there are already many threads and pools already. I
would really like to avoid having yet another Thread pool (creating thread pools should be
an extremely careful decision).
      The Akka thread pool executor is quite over-provisioned for the few actors we actually
use. I think it is perfectly feasible to use that one for the few extra futures introduced
here. In any case, if not reusing the Akka executor pool, then the thread pool needs to be
shut down in the TaskManager runner. Otherwise it creates a leak when running successive local
Flink jobs.
      - I am a bit consumed about the `SlotEnvironment`. Maybe it is mainly the name, but
what does it have to do with the slots? Is it not more like a network-messages specific *JobManager
      - The `ResultPartitionConsumableNotifier` could be per `Task` - that way, future multi-JobManager
assiciations would work seamlessly and it could directly call `fail(...)` on the Task without
having to go through the `TaskManager`. It could probably leave the TaskManager out of the
picture completely.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message