carbondata-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jack...@apache.org
Subject [13/50] [abbrv] carbondata git commit: [CARBONDATA-1278] Data Mismatch issue when dictionary column filter values doesn't exists in dictionary
Date Tue, 10 Oct 2017 03:08:00 GMT
[CARBONDATA-1278] Data Mismatch issue when dictionary column filter values doesn't exists in
dictionary

Root cause: when filter value is not present in dictionary, end key for that column is updated
with 0,and hence btree jump is not selecting all the leaf node.
Fix:
To handle this issue end key should be updated with Integer max value, so that it can go till
last leafnode of btree

This closes #1383


Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/d5dd78af
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/d5dd78af
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/d5dd78af

Branch: refs/heads/streaming_ingest
Commit: d5dd78afd237f10593c4403d9bf1d3bc482d3636
Parents: 6627cac
Author: kumarvishal <kumarvishal.1802@gmail.com>
Authored: Mon Sep 25 18:15:20 2017 +0530
Committer: Ravindra Pesala <ravi.pesala@gmail.com>
Committed: Wed Sep 27 14:12:00 2017 +0530

----------------------------------------------------------------------
 .../apache/carbondata/core/scan/filter/FilterUtil.java  | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/carbondata/blob/d5dd78af/core/src/main/java/org/apache/carbondata/core/scan/filter/FilterUtil.java
----------------------------------------------------------------------
diff --git a/core/src/main/java/org/apache/carbondata/core/scan/filter/FilterUtil.java b/core/src/main/java/org/apache/carbondata/core/scan/filter/FilterUtil.java
index 497ca8c..e8488f9 100644
--- a/core/src/main/java/org/apache/carbondata/core/scan/filter/FilterUtil.java
+++ b/core/src/main/java/org/apache/carbondata/core/scan/filter/FilterUtil.java
@@ -1066,12 +1066,16 @@ public final class FilterUtil {
         continue;
       }
       int keyOrdinalOfDimensionFromCurrentBlock = dimensionFromCurrentBlock.getKeyOrdinal();
+      int endFilterValue = 0;
       for (ColumnFilterInfo info : values) {
         if (keyOrdinalOfDimensionFromCurrentBlock < endKey.length) {
-          if (endKey[keyOrdinalOfDimensionFromCurrentBlock] > info.getFilterList()
-              .get(info.getFilterList().size() - 1)) {
-            endKey[keyOrdinalOfDimensionFromCurrentBlock] =
-                info.getFilterList().get(info.getFilterList().size() - 1);
+          endFilterValue = info.getFilterList().get(info.getFilterList().size() - 1);
+          if (endFilterValue == 0) {
+            endFilterValue =
+                segmentProperties.getDimColumnsCardinality()[keyOrdinalOfDimensionFromCurrentBlock];
+          }
+          if (endKey[keyOrdinalOfDimensionFromCurrentBlock] > endFilterValue) {
+            endKey[keyOrdinalOfDimensionFromCurrentBlock] = endFilterValue;
           }
         }
       }


Mime
View raw message