spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheng, Hao" <hao.ch...@intel.com>
Subject RE: CREATE TABLE AS SELECT does not work with temp tables in 1.2.0
Date Sun, 07 Dec 2014 04:28:07 GMT
I've created(reused) the PR https://github.com/apache/spark/pull/3336, hopefully we can fix
this regression.

Thanks for the reporting.

Cheng Hao

-----Original Message-----
From: Michael Armbrust [mailto:michael@databricks.com] 
Sent: Saturday, December 6, 2014 4:51 AM
To: kb
Cc: dev@spark.incubator.apache.org; Cheng Hao
Subject: Re: CREATE TABLE AS SELECT does not work with temp tables in 1.2.0

Thanks for reporting.  This looks like a regression related to:
https://github.com/apache/spark/pull/2570

I've filed it here: https://issues.apache.org/jira/browse/SPARK-4769

On Fri, Dec 5, 2014 at 12:03 PM, kb <kendb15@hotmail.com> wrote:

> I am having trouble getting "create table as select" or saveAsTable 
> from a hiveContext to work with temp tables in spark 1.2.  No issues 
> in 1.1.0 or
> 1.1.1
>
> Simple modification to test case in the hive SQLQuerySuite.scala:
>
> test("double nested data") {
>     sparkContext.parallelize(Nested1(Nested2(Nested3(1))) ::
> Nil).registerTempTable("nested")
>     checkAnswer(
>       sql("SELECT f1.f2.f3 FROM nested"),
>       1)
>     checkAnswer(sql("CREATE TABLE test_ctas_1234 AS SELECT * from 
> nested"),
> Seq.empty[Row])
>     checkAnswer(
>       sql("SELECT * FROM test_ctas_1234"),
>       sql("SELECT * FROM nested").collect().toSeq)
>   }
>
>
> output:
>
> 11:57:15.974 ERROR org.apache.hadoop.hive.ql.parse.SemanticAnalyzer:
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:45 Table not 
> found 'nested'
>         at
>
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1243)
>         at
>
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1192)
>         at
>
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9209)
>         at
>
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>         at
>
> org.apache.spark.sql.hive.execution.CreateTableAsSelect.metastoreRelation$lzycompute(CreateTableAsSelect.scala:59)
>         at
>
> org.apache.spark.sql.hive.execution.CreateTableAsSelect.metastoreRelation(CreateTableAsSelect.scala:55)
>         at
>
> org.apache.spark.sql.hive.execution.CreateTableAsSelect.sideEffectResult$lzycompute(CreateTableAsSelect.scala:82)
>         at
>
> org.apache.spark.sql.hive.execution.CreateTableAsSelect.sideEffectResult(CreateTableAsSelect.scala:70)
>         at
>
> org.apache.spark.sql.hive.execution.CreateTableAsSelect.execute(CreateTableAsSelect.scala:89)
>         at
>
> org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:425)
>         at
> org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:425)
>         at
> org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58)
>         at org.apache.spark.sql.SchemaRDD.<init>(SchemaRDD.scala:105)
>         at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:103)
>         at
>
> org.apache.spark.sql.hive.execution.SQLQuerySuite$$anonfun$4.apply$mcV$sp(SQLQuerySuite.scala:122)
>         at
>
> org.apache.spark.sql.hive.execution.SQLQuerySuite$$anonfun$4.apply(SQLQuerySuite.scala:117)
>         at
>
> org.apache.spark.sql.hive.execution.SQLQuerySuite$$anonfun$4.apply(SQLQuerySuite.scala:117)
>         at
>
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>         at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>         at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>         at org.scalatest.Transformer.apply(Transformer.scala:22)
>         at org.scalatest.Transformer.apply(Transformer.scala:20)
>         at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
>         at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
>         at org.scalatest.FunSuite.withFixture(FunSuite.scala:1555)
>         at
>
> org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
>         at
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>         at
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
>         at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
>         at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
>         at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
>         at
>
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>         at
>
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
>         at
>
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
>         at
>
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
>         at scala.collection.immutable.List.foreach(List.scala:318)
>         at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
>         at
> org.scalatest.SuperEngine.org
> $scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
>         at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
>         at
> org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
>         at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
>         at org.scalatest.Suite$class.run(Suite.scala:1424)
>         at
> org.scalatest.FunSuite.org
> $scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
>         at
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
>         at
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
>         at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
>         at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
>         at org.scalatest.FunSuite.run(FunSuite.scala:1555)
>         at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55)
>         at
>
> org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2563)
>         at
>
> org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2557)
>         at scala.collection.immutable.List.foreach(List.scala:318)
>         at
> org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2557)
>         at
>
> org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044)
>         at
>
> org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043)
>         at
>
> org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722)
>         at
>
> org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043)
>         at org.scalatest.tools.Runner$.run(Runner.scala:883)
>         at org.scalatest.tools.Runner.run(Runner.scala)
>         at
>
> org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:141)
>         at
>
> org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:32)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/CREATE-TABLE
> -AS-SELECT-does-not-work-with-temp-tables-in-1-2-0-tp9662.html
> Sent from the Apache Spark Developers List mailing list archive at 
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org For 
> additional commands, e-mail: dev-help@spark.apache.org
>
>
Mime
View raw message