tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "CruiseCheung (Jira)" <j...@apache.org>
Subject [jira] [Created] (TEZ-4301) TezChlid OOM cause PipelineSorter may create too many sort span
Date Tue, 30 Mar 2021 03:47:00 GMT
CruiseCheung created TEZ-4301:
---------------------------------

             Summary: TezChlid OOM cause PipelineSorter may create too many sort span
                 Key: TEZ-4301
                 URL: https://issues.apache.org/jira/browse/TEZ-4301
             Project: Apache Tez
          Issue Type: Bug
    Affects Versions: 0.9.2, 0.9.0
         Environment: Previous sort span:
[TezChild] |impl.PipelinedSorter|: Reducer 24: reserved.remaining()=965738496, reserved.metasize=16777216
[TezChild] |impl.PipelinedSorter|: Reducer 24: Span0.length = 677, perItem = 1400122
[TezChild] |impl.PipelinedSorter|: Reducer 24: done sorting span=0, length=677, time=0
[TezChild] |impl.PipelinedSorter|: Reducer 24: Heap = SpanIterator<0:676> (span=Span[16777216,947882594]),


[TezChild] |impl.PipelinedSorter|: Reducer 24: Span131.length = 1, perItem = 442
[TezChild] |impl.PipelinedSorter|: Reducer 24: reserved.remaining()=276247288, reserved.metasize=16
[TezChild] |impl.PipelinedSorter|: Reducer 24: New Span132.length = 1, perItem = 442, counter:533341
 [TezChild] |impl.PipelinedSorter|: Reducer 24: Span132.length = 1, perItem = 416
 [TezChild] |impl.PipelinedSorter|: Reducer 24: reserved.remaining()=276246856, reserved.metasize=16
 [TezChild] |impl.PipelinedSorter|: Reducer 24: New Span133.length = 1, perItem = 416, counter:533342

            Reporter: CruiseCheung


When previous span perItem is  much large than others, reserved.remaining() is large。This 
may create too many min sort span,cause now we use perItem length to deduce next sort span's
metaSize.

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message