carbondata-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From xuchuan...@apache.org
Subject carbondata git commit: [CARBONDATA-3068] fixed cannot load data from hdfs files without hdfs prefix
Date Fri, 02 Nov 2018 01:11:16 GMT
Repository: carbondata
Updated Branches:
  refs/heads/master ac94dbadc -> 269f4c378


[CARBONDATA-3068] fixed cannot load data from hdfs files without hdfs prefix

sql:
LOAD DATA INPATH '/tmp/test.csv' INTO TABLE test OPTIONS('QUOTECHAR'='"','TIMESTAMPFORMAT'='yyyy/MM/dd
HH:mm:ss');
error:
org.apache.carbondata.processing.exception.DataLoadingException: The input file does not exist:
/tmp/test.csv (state=,code=0)
but the file "test.csv" is in hdfs path, and hadoop conf "core-site.xml" has the property:

fs.defaultFS
hdfs://master:9000

This closes #2891


Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/269f4c37
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/269f4c37
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/269f4c37

Branch: refs/heads/master
Commit: 269f4c378bad1294430b7d1662469e300f0e04ed
Parents: ac94dba
Author: Sssan520 <liangap2008@aliyun.com>
Authored: Mon Jul 2 19:12:24 2018 +0800
Committer: xuchuanyin <xuchuanyin@hust.edu.cn>
Committed: Fri Nov 2 09:10:35 2018 +0800

----------------------------------------------------------------------
 .../src/main/scala/org/apache/spark/util/FileUtils.scala           | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/carbondata/blob/269f4c37/integration/spark-common/src/main/scala/org/apache/spark/util/FileUtils.scala
----------------------------------------------------------------------
diff --git a/integration/spark-common/src/main/scala/org/apache/spark/util/FileUtils.scala
b/integration/spark-common/src/main/scala/org/apache/spark/util/FileUtils.scala
index db63f6e..3c7fbbb 100644
--- a/integration/spark-common/src/main/scala/org/apache/spark/util/FileUtils.scala
+++ b/integration/spark-common/src/main/scala/org/apache/spark/util/FileUtils.scala
@@ -75,7 +75,7 @@ object FileUtils {
       val filePaths = inputPath.split(",")
       for (i <- 0 until filePaths.size) {
         val filePath = CarbonUtil.checkAndAppendHDFSUrl(filePaths(i))
-        val carbonFile = FileFactory.getCarbonFile(filePaths(i), hadoopConf)
+        val carbonFile = FileFactory.getCarbonFile(filePath, hadoopConf)
         if (!carbonFile.exists()) {
           throw new DataLoadingException(
             s"The input file does not exist: ${CarbonUtil.removeAKSK(filePaths(i))}" )


Mime
View raw message