tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajesh Balamohan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (TEZ-3769) Unordered: Fix wrong stats being sent out in the last event when final merge is disabled
Date Thu, 22 Jun 2017 11:47:00 GMT
Rajesh Balamohan created TEZ-3769:
-------------------------------------

             Summary: Unordered: Fix wrong stats being sent out in the last event when final
merge is disabled
                 Key: TEZ-3769
                 URL: https://issues.apache.org/jira/browse/TEZ-3769
             Project: Apache Tez
          Issue Type: Bug
            Reporter: Rajesh Balamohan


When final merge is disabled (without pipelining), wrong stats was sent out in the last event.


It was based on {{numRecordsPerPartition}} which contains the overall partition data. It should
be ideally be based on the spill result and its buffers.

Also, {{finalSpill}} was unncessarily sending events when no data was present (i.e, when currentBuffer
didn't have any data).  This can be optimized to reduce the number of events being sent across.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message