carbondata-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From indhumuthumurug...@apache.org
Subject [carbondata] branch master updated: [CARBONDATA-4094]: Fix fallback count(*) issue on partition table with index server
Date Tue, 22 Dec 2020 13:38:34 GMT
This is an automated email from the ASF dual-hosted git repository.

indhumuthumurugesh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git


The following commit(s) were added to refs/heads/master by this push:
     new 1dfcdec  [CARBONDATA-4094]: Fix fallback count(*) issue on partition table with index
server
1dfcdec is described below

commit 1dfcdec6311e48d9a330c8a4aec8a5592106adba
Author: Vikram Ahuja <vikramahuja8803@gmail.com>
AuthorDate: Mon Dec 21 13:30:11 2020 +0530

    [CARBONDATA-4094]: Fix fallback count(*) issue on partition table with index server
    
    Why is this PR needed?
    The used asJava converts to java "in place", without copying the whole data to save
    time and memory and it just simply wraps the scala collection with a class that
    conforms to the java interface and thus java serializer is not able to serialize it.
    
    What changes were proposed in this PR?
    Converting it to list, so that it is able to serialize a list.
    
    Does this PR introduce any user interface change?
    No
    
    Is any new testcase added?
    No
    
    This closes #4061
---
 .../main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java | 4 ++++
 .../spark/src/main/scala/org/apache/spark/sql/CarbonCountStar.scala   | 2 +-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java
b/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java
index 9765913..4d917fb 100644
--- a/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java
+++ b/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java
@@ -404,6 +404,10 @@ public abstract class CarbonInputFormat<T> extends FileInputFormat<Void,
T> {
       }
       return indexJob.executeCountJob(indexInputFormat, configuration);
     } catch (Exception e) {
+      if (CarbonProperties.getInstance().isFallBackDisabled()) {
+        LOG.error("Fallback is disabled");
+        throw e;
+      }
       LOG.error("Failed to get count from index server. Initializing fallback", e);
       IndexJob indexJob = IndexUtil.getEmbeddedJob();
       return indexJob.executeCountJob(indexInputFormat, configuration);
diff --git a/integration/spark/src/main/scala/org/apache/spark/sql/CarbonCountStar.scala b/integration/spark/src/main/scala/org/apache/spark/sql/CarbonCountStar.scala
index 6ca124f..678a0be 100644
--- a/integration/spark/src/main/scala/org/apache/spark/sql/CarbonCountStar.scala
+++ b/integration/spark/src/main/scala/org/apache/spark/sql/CarbonCountStar.scala
@@ -63,7 +63,7 @@ case class CarbonCountStar(
           sparkSession,
           TableIdentifier(
             carbonTable.getTableName,
-            Some(carbonTable.getDatabaseName))).map(_.asJava).orNull, false),
+            Some(carbonTable.getDatabaseName))).map(_.toList.asJava).orNull, false),
       carbonTable)
 
     if (CarbonProperties.isQueryStageInputEnabled) {


Mime
View raw message