carbondata-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject [carbondata] branch master updated: [CARBONDATA-3871][FOLLOW-UP] Fix memory issue of loading sort step when get data from row page
Date Tue, 21 Jul 2020 03:40:27 GMT
This is an automated email from the ASF dual-hosted git repository.

qiangcai pushed a commit to branch master
in repository

The following commit(s) were added to refs/heads/master by this push:
     new c5c1f59  [CARBONDATA-3871][FOLLOW-UP] Fix memory issue of loading sort step when
get data from row page
c5c1f59 is described below

commit c5c1f597737f3bd341eb4088ba530b5a81ece89c
Author: Manhua <>
AuthorDate: Wed Jul 15 17:08:11 2020 +0800

    [CARBONDATA-3871][FOLLOW-UP] Fix memory issue of loading sort step when get data from
row page
    Why is this PR needed?
    PR #3804 free the row page after running in-memory intermeidate merge, but the data will
still be used in final sort because that merge only acquire the memory address and row page
index instead of moving the rows from here to there.
    What changes were proposed in this PR?
    Remove code for freeing the row page in case of in-memory intermeidate merge.
    As a special case, add comment for explanation.
    Does this PR introduce any user interface change?
    Is any new testcase added?
    This closes #3845
 .../sort/unsafe/merger/     | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/
index 558a40a..f562b90 100644
--- a/processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/
+++ b/processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/
@@ -147,7 +147,10 @@ public class UnsafeInMemoryIntermediateDataMerger implements Callable<Void>
     // check if there no entry present
     if (!poll.hasNext()) {
-      poll.close();
+      // do not close the poll to free the memory of rowPage here
+      // because data in memory will be used in final merge sort
+      // and close in `UnsafeFinalMergePageHolder` in case of `spillDisk`
+      // is false. Please refer method `writeDataToMemory` in this class
       // change the file counter

View raw message