carbondata-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jack...@apache.org
Subject carbondata git commit: [CARBONDATA-2464]Fixed OOM issue in case of Complex type
Date Thu, 10 May 2018 07:27:39 GMT
Repository: carbondata
Updated Branches:
  refs/heads/master cc0cbbac7 -> ceb7c8dd1


[CARBONDATA-2464]Fixed OOM issue in case of Complex type

Problem: Query with Complex type is failing with OOM

Root Cause: Complex type child column(No-dictionary) values are written in LV format, while
reading the data it will read length then based on length it is reading the data. Converting
byte array to int is giving wrong length value, because of this it's trying to create big
memory chunk and as memory is not available in Unsafe it is failing with OOM.

Code issue: While converting byte array to int it is not masking the the byte values and because
of this is giving wrong integer value.

Solution: Mask each byte and then left shift the bits

This closes #2288


Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/ceb7c8dd
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/ceb7c8dd
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/ceb7c8dd

Branch: refs/heads/master
Commit: ceb7c8dd1ced457c7ce34f016abf30102e4931a9
Parents: cc0cbba
Author: kumarvishal09 <kumarvishal1802@gmail.com>
Authored: Wed May 9 17:04:21 2018 +0530
Committer: Jacky Li <jacky.likun@qq.com>
Committed: Thu May 10 15:27:32 2018 +0800

----------------------------------------------------------------------
 .../main/java/org/apache/carbondata/core/util/ByteUtil.java    | 4 ++--
 .../java/org/apache/carbondata/core/util/ByteUtilTest.java     | 6 ++++++
 2 files changed, 8 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/carbondata/blob/ceb7c8dd/core/src/main/java/org/apache/carbondata/core/util/ByteUtil.java
----------------------------------------------------------------------
diff --git a/core/src/main/java/org/apache/carbondata/core/util/ByteUtil.java b/core/src/main/java/org/apache/carbondata/core/util/ByteUtil.java
index d1c16bb..52fc3c3 100644
--- a/core/src/main/java/org/apache/carbondata/core/util/ByteUtil.java
+++ b/core/src/main/java/org/apache/carbondata/core/util/ByteUtil.java
@@ -517,8 +517,8 @@ public final class ByteUtil {
   }
 
   public static int toInt(byte[] bytes, int offset) {
-    return (((int)bytes[offset]) << 24) + (((int)bytes[offset + 1]) << 16) +
-        (((int)bytes[offset + 2]) << 8) + bytes[offset + 3];
+    return (((int)bytes[offset] & 0xff) << 24) + (((int)bytes[offset + 1] &
0xff) << 16) +
+        (((int)bytes[offset + 2] & 0xff) << 8) + ((int)bytes[offset + 3] &
0xff);
   }
 
   public static void setInt(byte[] data, int offset, int value) {

http://git-wip-us.apache.org/repos/asf/carbondata/blob/ceb7c8dd/core/src/test/java/org/apache/carbondata/core/util/ByteUtilTest.java
----------------------------------------------------------------------
diff --git a/core/src/test/java/org/apache/carbondata/core/util/ByteUtilTest.java b/core/src/test/java/org/apache/carbondata/core/util/ByteUtilTest.java
index d51e184..d93aa49 100644
--- a/core/src/test/java/org/apache/carbondata/core/util/ByteUtilTest.java
+++ b/core/src/test/java/org/apache/carbondata/core/util/ByteUtilTest.java
@@ -113,6 +113,12 @@ public class ByteUtilTest extends TestCase {
         prepareBuffers();
         assertFalse(UnsafeComparer.INSTANCE.compareTo(buff1, buff2) < 0);
     }
+    @Test
+    public void testIntConversion() {
+        byte[] data = new byte[4];
+        ByteUtil.setInt(data, 0, 968);
+        assertEquals(ByteUtil.toInt(data, 0), 968);
+    }
 
     @Test
     public void testEqualToCase() {


Mime
View raw message