spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-4943) Parsing error for query with table name having dot
Date Mon, 05 Jan 2015 22:14:34 GMT

    [ https://issues.apache.org/jira/browse/SPARK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265212#comment-14265212
] 

Alex Liu commented on SPARK-4943:
---------------------------------

Catalog part of table identifier should be handled by Spark SQL which calls the registered
catalog Context to connect to the underline datasources. cluster/database/scheme/table should
be handled by datasource(Cassandra Spark SQL integration can handle cluster, database and
table level join).

e.g.
{code}
SELECT test1.a, test1.b, test2.c FROM cassandra.cluster.database.table1 AS test1
    LEFT OUTER JOIN mySql.cluster.database.table2 AS test2 ON test1.a = test2.a
{code}

so cluster.database.table1 is passed to cassandra catalog datasource, cassandra is handled
by Spark SQL to call cassandraContext which then call the underline datasource.

cluster.database.table2 is passed to mySql catalog datasource, mySql is handled by Spark SQL
to call the mySqlContext which then call the underline datasource.


If USE command is used, then all tableIdentifiers are passed to datasource.  e.g.
{code}
USE cassandra
SELECT test1.a, test1.b, test2.c FROM cluster1.database.table AS test1
    LEFT OUTER JOIN cluster2.database.table AS test2 ON test1.a = test2.a
{code}

cluster1.database.table1 and cluster2.database.table are passed to cassandra datasource



> Parsing error for query with table name having dot
> --------------------------------------------------
>
>                 Key: SPARK-4943
>                 URL: https://issues.apache.org/jira/browse/SPARK-4943
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.2.0
>            Reporter: Alex Liu
>
> When integrating Spark 1.2.0 with Cassandra SQL, the following query is broken. It was
working for Spark 1.1.0 version. Basically we use the table name having dot to include database
name 
> {code}
> [info]   java.lang.RuntimeException: [1.29] failure: ``UNION'' expected but `.' found
> [info] 
> [info] SELECT test1.a FROM sql_test.test1 AS test1 UNION DISTINCT SELECT test2.a FROM
sql_test.test2 AS test2
> [info]                             ^
> [info]   at scala.sys.package$.error(package.scala:27)
> [info]   at org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:33)
> [info]   at org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info]   at org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info]   at org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:174)
> [info]   at org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:173)
> [info]   at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
> [info]   at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
> [info]   at scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info]   at scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info]   at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info]   at scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info]   at scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info]   at scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
> [info]   at scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info]   at scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info]   at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info]   at scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info]   at scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info]   at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> [info]   at scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
> [info]   at scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
> [info]   at org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:31)
> [info]   at org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info]   at org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info]   at scala.Option.getOrElse(Option.scala:120)
> [info]   at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:83)
> [info]   at org.apache.spark.sql.cassandra.CassandraSQLContext.cassandraSql(CassandraSQLContext.scala:53)
> [info]   at org.apache.spark.sql.cassandra.CassandraSQLContext.sql(CassandraSQLContext.scala:56)
> [info]   at com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply$mcV$sp(CassandraSQLSpec.scala:169)
> [info]   at com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info]   at com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info]   at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
> [info]   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> [info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> [info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
> [info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
> [info]   at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1647)
> [info]   at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
> [info]   at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1683)
> [info]   at org.scalatest.FlatSpecLike$class.invokeWithFixture$1(FlatSpecLike.scala:1644)
> [info]   at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656)
> [info]   at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656)
> [info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
> [info]   at org.scalatest.FlatSpecLike$class.runTest(FlatSpecLike.scala:1656)
> [info]   at org.scalatest.FlatSpec.runTest(FlatSpec.scala:1683)
> [info]   at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714)
> [info]   at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714)
> [info]   at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
> [info]   at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
> [info]   at scala.collection.immutable.List.foreach(List.scala:318)
> [info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
> [info]   at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
> [info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
> [info]   at org.scalatest.FlatSpecLike$class.runTests(FlatSpecLike.scala:1714)
> [info]   at org.scalatest.FlatSpec.runTests(FlatSpec.scala:1683)
> [info]   at org.scalatest.Suite$class.run(Suite.scala:1424)
> [info]   at org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:1683)
> [info]   at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1760)
> [info]   at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1760)
> [info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
> [info]   at org.scalatest.FlatSpecLike$class.run(FlatSpecLike.scala:1760)
> [info]   at org.scalatest.FlatSpec.run(FlatSpec.scala:1683)
> [info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:466)
> [info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:677)
> [info]   at sbt.ForkMain$Run$2.call(ForkMain.java:294)
> [info]   at sbt.ForkMain$Run$2.call(ForkMain.java:284)
> [info]   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> [info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [info]   at java.lang.Thread.run(Thread.java:745)
> [info] - should allow to select rows with union distinct clause *** FAILED *** (46 milliseconds)
> [info]   java.lang.RuntimeException: [1.29] failure: ``UNION'' expected but `.' found
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message