tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Eagles (JIRA)" <j...@apache.org>
Subject [jira] [Created] (TEZ-3954) Reduce Tez Shuffle Handler Memory needs for holding TezIndexRecords
Date Sat, 16 Jun 2018 05:11:00 GMT
Jonathan Eagles created TEZ-3954:
------------------------------------

             Summary: Reduce Tez Shuffle Handler Memory needs for holding TezIndexRecords
                 Key: TEZ-3954
                 URL: https://issues.apache.org/jira/browse/TEZ-3954
             Project: Apache Tez
          Issue Type: Bug
            Reporter: Jonathan Eagles
            Assignee: Jonathan Eagles


Tez Shuffle Handler holds on to entire SpillRecord to better accommodate for range fetch operation.
For single reduce fetch holding on to spillRecord can overwhelm memory if Index Cache is overrun.
In the case of single fetch, only keep the single tez index record needed and for range fetch,
keep the entire spill record in memory, but free it after it is needed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message