tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Created] (TEZ-3217) Optimize DataMovementEvent routing between AM and downstream vertex
Date Sat, 16 Apr 2016 03:22:25 GMT
Ming Ma created TEZ-3217:
----------------------------

             Summary: Optimize DataMovementEvent routing between AM and downstream vertex
                 Key: TEZ-3217
                 URL: https://issues.apache.org/jira/browse/TEZ-3217
             Project: Apache Tez
          Issue Type: Improvement
            Reporter: Ming Ma


Follow up on TEZ-3206 discussion, we might be able to optimize the DME routing from AM to
downstream vertex, mostly for the auto-parallelism case.

* DME's empty partition payload has all empty partitions from a specific mapper. At the reducer
side, it only cares about the partitions it is responsible for, not partitions belong to other
reducers. Perhaps we can optimize AM to send only the relevant empty partitions to that reducer.
* Instead of sending one DME to a given reducer at a time, it can batch all DMEs belonging
to a given (mapper, reducer) pair with the common empty partition payload, similar to CompositeDataMovementEvent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message