tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yingda Chen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (TEZ-3999) Extend VertexManagerPlugin interface to allow for relevant events notification
Date Fri, 05 Oct 2018 19:34:00 GMT
Yingda Chen created TEZ-3999:

             Summary: Extend VertexManagerPlugin interface to allow for relevant events notification
                 Key: TEZ-3999
                 URL: https://issues.apache.org/jira/browse/TEZ-3999
             Project: Apache Tez
          Issue Type: Task
            Reporter: Yingda Chen

This is an umbrella task of TEZ-3997

**For concurrent connection, the downstream and upstream vertices would be running concurrently,
and in some cases, they would be scheduled at the same time as well, such as (sub-graph) gang
scheduling. However, *this is not always true*. In the example in Fig. 1, tasks in PS vertex
should be running before tasks in W vertex should be scheduled. Since otherwise if the resource
requests for PS cannot be fulfilled first, W will be spinning in vain. In other examples,
as long as part of tasks in upstream vertex are running, we can start scheduling downstream


In other words, if we put this into the context of existing interface/implementation of VertexMangerPlugin,
we can see strong duality of “OnSourceTaskRunning” for concurrent connection vs the “OnSourceTaskCompleted”
for (existing) sequential connection. Therefore, we propose an addition of “_onConcurrentSourceTaskRunning(TaskAttemptIdentifer
attempt)_” interface to the VertexManager Plugin, with default implementation being not

This change will also include the logic to add source task running event and to send such
events to downstream vertices. To reduce unnecessary event traffic, we will limit the sending
of such events to CONCURRENT edge, and when the ConcurrentSchedulingType is specified to be

This message was sent by Atlassian JIRA

View raw message