tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yingdachen <...@git.apache.org>
Subject [GitHub] tez issue #33: TEZ-3998: support constructing DAG with concurrent edge
Date Fri, 19 Oct 2018 12:21:08 GMT
Github user yingdachen commented on the issue:

    @jteagles thanks for your input.
    Regarding the implication of RM(YARN) supporting gang scheduling, it largely depends on
how application/runtime is implemented, and may not be an absolute necessity.  While it would
be best if strong gang-scheduling can be guaranteed by RM, but even when without, AM can still
assist in several ways:
    1. AM can schedule all tasks in upstream and downstream vertices simultaneously, as is
implemented in this change
    2. AM can gather information about downstream/upstream tasks and inform dependent tasks
to put things into action only when all dependencies are satisfied. This will require mechanisms
such as waiting/timeout/retry to be implemented in application runtime, but that would be
something needed anyways, even with RM gang-scheduling.  Some of the changes along this line
would be covered by TEZ-4000.


View raw message