carbondata-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From chenliang...@apache.org
Subject [18/50] [abbrv] incubator-carbondata git commit: [Bug]fix exception thrown when all data is bad records (#755)
Date Thu, 30 Jun 2016 17:42:05 GMT
[Bug]fix exception thrown  when all data is bad records (#755)

1. In case all the records go to bad records in in surrogate key gen step, then also level
cardinality file is getting written. Fact file writer gets initialized when level cardinality
file is written and after that the process fails as there is no actual data leading to data
load failure.
2. The sort key step throws null pinter exception as empty object row which has null value
is put from surrogate step.

Fix:
1. Write level cardinality file only if there is at least one valid row in data.
2. If there is no valid data, then do call putRow method.

Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/f16eb54f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/f16eb54f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/f16eb54f

Branch: refs/heads/master
Commit: f16eb54f9527622249bb47a2c101c91f22c1ef5f
Parents: 5b6081d
Author: manishgupta88 <tomanishgupta18@gmail.com>
Authored: Mon Jun 27 10:29:24 2016 +0800
Committer: david <QiangCai@users.noreply.github.com>
Committed: Mon Jun 27 10:29:24 2016 +0800

----------------------------------------------------------------------
 .../csvbased/CarbonCSVBasedSeqGenStep.java                     | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/f16eb54f/processing/src/main/java/org/carbondata/processing/surrogatekeysgenerator/csvbased/CarbonCSVBasedSeqGenStep.java
----------------------------------------------------------------------
diff --git a/processing/src/main/java/org/carbondata/processing/surrogatekeysgenerator/csvbased/CarbonCSVBasedSeqGenStep.java
b/processing/src/main/java/org/carbondata/processing/surrogatekeysgenerator/csvbased/CarbonCSVBasedSeqGenStep.java
index 39335d1..d5c8e61 100644
--- a/processing/src/main/java/org/carbondata/processing/surrogatekeysgenerator/csvbased/CarbonCSVBasedSeqGenStep.java
+++ b/processing/src/main/java/org/carbondata/processing/surrogatekeysgenerator/csvbased/CarbonCSVBasedSeqGenStep.java
@@ -457,15 +457,15 @@ public class CarbonCSVBasedSeqGenStep extends BaseStep {
       }
 
       startReadingProcess(numberOfNodes);
-      CarbonUtil.writeLevelCardinalityFile(loadFolderLoc, meta.getTableName(),
-          getUpdatedCardinality());
       badRecordslogger.closeStreams();
       if (!meta.isAggregate()) {
         closeNormalizedHierFiles();
       }
       if (writeCounter == 0) {
-        putRow(data.getOutputRowMeta(), new Object[outSize]);
+        return processWhenRowIsNull();
       }
+      CarbonUtil.writeLevelCardinalityFile(loadFolderLoc, meta.getTableName(),
+          getUpdatedCardinality());
       LOGGER.info("Record Procerssed For table: " + meta.getTableName());
       String logMessage =
           "Summary: Carbon CSV Based Seq Gen Step : " + readCounter + ": Write: " + writeCounter;


Mime
View raw message