spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haopu Wang" <HW...@qilinsoft.com>
Subject [SparkSQL] Reuse HiveContext to different Hive warehouse?
Date Tue, 10 Mar 2015 10:37:34 GMT
I'm using Spark 1.3.0 RC3 build with Hive support.

 

In Spark Shell, I want to reuse the HiveContext instance to different
warehouse locations. Below are the steps for my test (Assume I have
loaded a file into table "src").

 

======

15/03/10 18:22:59 INFO SparkILoop: Created sql context (with Hive
support)..

SQL context available as sqlContext.

scala> sqlContext.sql("SET hive.metastore.warehouse.dir=/test/w")

scala> sqlContext.sql("SELECT * from src").saveAsTable("table1")

scala> sqlContext.sql("SET hive.metastore.warehouse.dir=/test/w2")

scala> sqlContext.sql("SELECT * from src").saveAsTable("table2")

======

After these steps, the tables are stored in "/test/w" only. I expect
"table2" to be stored in "/test/w2" folder.

 

Another question is: if I set "hive.metastore.warehouse.dir" to a HDFS
folder, I cannot use saveAsTable()? Is this by design? Exception stack
trace is below:

======

15/03/10 18:35:28 INFO BlockManagerMaster: Updated info of block
broadcast_0_piece0

15/03/10 18:35:28 INFO SparkContext: Created broadcast 0 from broadcast
at TableReader.scala:74

java.lang.IllegalArgumentException: Wrong FS:
hdfs://server:8020/space/warehouse/table2, expected: file:///

        at
org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643)

        at
org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:463)

        at
org.apache.hadoop.fs.FilterFileSystem.makeQualified(FilterFileSystem.jav
a:118)

        at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$6.a
pply(newParquet.scala:252)

        at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$6.a
pply(newParquet.scala:251)

        at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sc
ala:244)

        at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sc
ala:244)

        at scala.collection.immutable.List.foreach(List.scala:318)

        at
scala.collection.TraversableLike$class.map(TraversableLike.scala:244)

        at
scala.collection.AbstractTraversable.map(Traversable.scala:105)

        at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache.refresh(newP
arquet.scala:251)

        at
org.apache.spark.sql.parquet.ParquetRelation2.<init>(newParquet.scala:37
0)

        at
org.apache.spark.sql.parquet.DefaultSource.createRelation(newParquet.sca
la:96)

        at
org.apache.spark.sql.parquet.DefaultSource.createRelation(newParquet.sca
la:125)

        at
org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:308)

        at
org.apache.spark.sql.hive.execution.CreateMetastoreDataSourceAsSelect.ru
n(commands.scala:217)

        at
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompu
te(commands.scala:55)

        at
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands
.scala:55)

        at
org.apache.spark.sql.execution.ExecutedCommand.execute(commands.scala:65
)

        at
org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLConte
xt.scala:1088)

        at
org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:10
88)

        at
org.apache.spark.sql.DataFrame.saveAsTable(DataFrame.scala:1048)

        at
org.apache.spark.sql.DataFrame.saveAsTable(DataFrame.scala:998)

        at
org.apache.spark.sql.DataFrame.saveAsTable(DataFrame.scala:964)

        at
org.apache.spark.sql.DataFrame.saveAsTable(DataFrame.scala:942)

        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)

        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)

        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)

        at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)

        at $iwC$$iwC$$iwC$$iwC.<init>(<console>:31)

        at $iwC$$iwC$$iwC.<init>(<console>:33)

        at $iwC$$iwC.<init>(<console>:35)

        at $iwC.<init>(<console>:37)

        at <init>(<console>:39)

 

Thank you very much!

 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message