Rajesh Balamohan created TEZ-3769:
-------------------------------------
Summary: Unordered: Fix wrong stats being sent out in the last event when final
merge is disabled
Key: TEZ-3769
URL: https://issues.apache.org/jira/browse/TEZ-3769
Project: Apache Tez
Issue Type: Bug
Reporter: Rajesh Balamohan
When final merge is disabled (without pipelining), wrong stats was sent out in the last event.
It was based on {{numRecordsPerPartition}} which contains the overall partition data. It should
be ideally be based on the spill result and its buffers.
Also, {{finalSpill}} was unncessarily sending events when no data was present (i.e, when currentBuffer
didn't have any data). This can be optimized to reduce the number of events being sent across.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
|