tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Created] (TEZ-3207) Have the unordered ShuffleManager support fetching multiple partitions from the same source task
Date Fri, 08 Apr 2016 22:36:25 GMT
Ming Ma created TEZ-3207:
----------------------------

             Summary: Have the unordered ShuffleManager support fetching multiple partitions
from the same source task
                 Key: TEZ-3207
                 URL: https://issues.apache.org/jira/browse/TEZ-3207
             Project: Apache Tez
          Issue Type: Bug
            Reporter: Ming Ma


The ordered grouped {{ShuffleScheduler}} can support fetching multiple partitions from the
same source task. But for the unordered ShuffleManager, it only supports one partition per
source task due to the following issue where {{identifier}} doesn't take partition id into
account.

{noformat}
  public void addKnownInput(String hostName, int port,
      InputAttemptIdentifier srcAttemptIdentifier, int srcPhysicalIndex) {
    String identifier = InputHost.createIdentifier(hostName, port);
    InputHost host = knownSrcHosts.get(identifier);
    ....
  }
{noformat}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message