spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <mich...@databricks.com>
Subject Re: CREATE TABLE AS SELECT does not work with temp tables in 1.2.0
Date Tue, 09 Dec 2014 01:44:53 GMT
This is merged now and should be fixed in the next 1.2 RC.

On Sat, Dec 6, 2014 at 8:28 PM, Cheng, Hao <hao.cheng@intel.com> wrote:

> I've created(reused) the PR https://github.com/apache/spark/pull/3336,
> hopefully we can fix this regression.
>
> Thanks for the reporting.
>
> Cheng Hao
>
> -----Original Message-----
> From: Michael Armbrust [mailto:michael@databricks.com]
> Sent: Saturday, December 6, 2014 4:51 AM
> To: kb
> Cc: dev@spark.incubator.apache.org; Cheng Hao
> Subject: Re: CREATE TABLE AS SELECT does not work with temp tables in 1.2.0
>
> Thanks for reporting.  This looks like a regression related to:
> https://github.com/apache/spark/pull/2570
>
> I've filed it here: https://issues.apache.org/jira/browse/SPARK-4769
>
> On Fri, Dec 5, 2014 at 12:03 PM, kb <kendb15@hotmail.com> wrote:
>
> > I am having trouble getting "create table as select" or saveAsTable
> > from a hiveContext to work with temp tables in spark 1.2.  No issues
> > in 1.1.0 or
> > 1.1.1
> >
> > Simple modification to test case in the hive SQLQuerySuite.scala:
> >
> > test("double nested data") {
> >     sparkContext.parallelize(Nested1(Nested2(Nested3(1))) ::
> > Nil).registerTempTable("nested")
> >     checkAnswer(
> >       sql("SELECT f1.f2.f3 FROM nested"),
> >       1)
> >     checkAnswer(sql("CREATE TABLE test_ctas_1234 AS SELECT * from
> > nested"),
> > Seq.empty[Row])
> >     checkAnswer(
> >       sql("SELECT * FROM test_ctas_1234"),
> >       sql("SELECT * FROM nested").collect().toSeq)
> >   }
> >
> >
> > output:
> >
> > 11:57:15.974 ERROR org.apache.hadoop.hive.ql.parse.SemanticAnalyzer:
> > org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:45 Table not
> > found 'nested'
> >         at
> >
> >
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1243)
> >         at
> >
> >
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1192)
> >         at
> >
> >
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9209)
> >         at
> >
> >
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> >         at
> >
> >
> org.apache.spark.sql.hive.execution.CreateTableAsSelect.metastoreRelation$lzycompute(CreateTableAsSelect.scala:59)
> >         at
> >
> >
> org.apache.spark.sql.hive.execution.CreateTableAsSelect.metastoreRelation(CreateTableAsSelect.scala:55)
> >         at
> >
> >
> org.apache.spark.sql.hive.execution.CreateTableAsSelect.sideEffectResult$lzycompute(CreateTableAsSelect.scala:82)
> >         at
> >
> >
> org.apache.spark.sql.hive.execution.CreateTableAsSelect.sideEffectResult(CreateTableAsSelect.scala:70)
> >         at
> >
> >
> org.apache.spark.sql.hive.execution.CreateTableAsSelect.execute(CreateTableAsSelect.scala:89)
> >         at
> >
> >
> org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:425)
> >         at
> >
> org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:425)
> >         at
> > org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58)
> >         at org.apache.spark.sql.SchemaRDD.<init>(SchemaRDD.scala:105)
> >         at
> org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:103)
> >         at
> >
> >
> org.apache.spark.sql.hive.execution.SQLQuerySuite$$anonfun$4.apply$mcV$sp(SQLQuerySuite.scala:122)
> >         at
> >
> >
> org.apache.spark.sql.hive.execution.SQLQuerySuite$$anonfun$4.apply(SQLQuerySuite.scala:117)
> >         at
> >
> >
> org.apache.spark.sql.hive.execution.SQLQuerySuite$$anonfun$4.apply(SQLQuerySuite.scala:117)
> >         at
> >
> >
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
> >         at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> >         at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> >         at org.scalatest.Transformer.apply(Transformer.scala:22)
> >         at org.scalatest.Transformer.apply(Transformer.scala:20)
> >         at
> org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
> >         at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
> >         at org.scalatest.FunSuite.withFixture(FunSuite.scala:1555)
> >         at
> >
> >
> org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
> >         at
> >
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
> >         at
> >
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
> >         at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
> >         at
> org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
> >         at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
> >         at
> >
> >
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
> >         at
> >
> >
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
> >         at
> >
> >
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
> >         at
> >
> >
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
> >         at scala.collection.immutable.List.foreach(List.scala:318)
> >         at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
> >         at
> > org.scalatest.SuperEngine.org
> > $scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
> >         at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
> >         at
> > org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
> >         at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
> >         at org.scalatest.Suite$class.run(Suite.scala:1424)
> >         at
> > org.scalatest.FunSuite.org
> > $scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
> >         at
> > org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
> >         at
> > org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
> >         at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
> >         at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
> >         at org.scalatest.FunSuite.run(FunSuite.scala:1555)
> >         at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55)
> >         at
> >
> >
> org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2563)
> >         at
> >
> >
> org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2557)
> >         at scala.collection.immutable.List.foreach(List.scala:318)
> >         at
> > org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2557)
> >         at
> >
> >
> org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044)
> >         at
> >
> >
> org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043)
> >         at
> >
> >
> org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722)
> >         at
> >
> >
> org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043)
> >         at org.scalatest.tools.Runner$.run(Runner.scala:883)
> >         at org.scalatest.tools.Runner.run(Runner.scala)
> >         at
> >
> >
> org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:141)
> >         at
> >
> >
> org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:32)
> >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >         at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >         at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >         at java.lang.reflect.Method.invoke(Method.java:606)
> >         at
> > com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> > http://apache-spark-developers-list.1001551.n3.nabble.com/CREATE-TABLE
> > -AS-SELECT-does-not-work-with-temp-tables-in-1-2-0-tp9662.html
> > Sent from the Apache Spark Developers List mailing list archive at
> > Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org For
> > additional commands, e-mail: dev-help@spark.apache.org
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message