From issues-return-22582-apmail-spark-issues-archive=spark.apache.org@spark.apache.org Mon Jan 5 22:14:33 2015 Return-Path: X-Original-To: apmail-spark-issues-archive@minotaur.apache.org Delivered-To: apmail-spark-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B6A6A172FD for ; Mon, 5 Jan 2015 22:14:33 +0000 (UTC) Received: (qmail 58422 invoked by uid 500); 5 Jan 2015 22:14:34 -0000 Delivered-To: apmail-spark-issues-archive@spark.apache.org Received: (qmail 58389 invoked by uid 500); 5 Jan 2015 22:14:34 -0000 Mailing-List: contact issues-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@spark.apache.org Received: (qmail 58379 invoked by uid 99); 5 Jan 2015 22:14:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jan 2015 22:14:34 +0000 Date: Mon, 5 Jan 2015 22:14:34 +0000 (UTC) From: "Alex Liu (JIRA)" To: issues@spark.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (SPARK-4943) Parsing error for query with table name having dot MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/SPARK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265212#comment-14265212 ] Alex Liu commented on SPARK-4943: --------------------------------- Catalog part of table identifier should be handled by Spark SQL which calls the registered catalog Context to connect to the underline datasources. cluster/database/scheme/table should be handled by datasource(Cassandra Spark SQL integration can handle cluster, database and table level join). e.g. {code} SELECT test1.a, test1.b, test2.c FROM cassandra.cluster.database.table1 AS test1 LEFT OUTER JOIN mySql.cluster.database.table2 AS test2 ON test1.a = test2.a {code} so cluster.database.table1 is passed to cassandra catalog datasource, cassandra is handled by Spark SQL to call cassandraContext which then call the underline datasource. cluster.database.table2 is passed to mySql catalog datasource, mySql is handled by Spark SQL to call the mySqlContext which then call the underline datasource. If USE command is used, then all tableIdentifiers are passed to datasource. e.g. {code} USE cassandra SELECT test1.a, test1.b, test2.c FROM cluster1.database.table AS test1 LEFT OUTER JOIN cluster2.database.table AS test2 ON test1.a = test2.a {code} cluster1.database.table1 and cluster2.database.table are passed to cassandra datasource > Parsing error for query with table name having dot > -------------------------------------------------- > > Key: SPARK-4943 > URL: https://issues.apache.org/jira/browse/SPARK-4943 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.2.0 > Reporter: Alex Liu > > When integrating Spark 1.2.0 with Cassandra SQL, the following query is broken. It was working for Spark 1.1.0 version. Basically we use the table name having dot to include database name > {code} > [info] java.lang.RuntimeException: [1.29] failure: ``UNION'' expected but `.' found > [info] > [info] SELECT test1.a FROM sql_test.test1 AS test1 UNION DISTINCT SELECT test2.a FROM sql_test.test2 AS test2 > [info] ^ > [info] at scala.sys.package$.error(package.scala:27) > [info] at org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:33) > [info] at org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79) > [info] at org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79) > [info] at org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:174) > [info] at org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:173) > [info] at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136) > [info] at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135) > [info] at scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242) > [info] at scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242) > [info] at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222) > [info] at scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254) > [info] at scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254) > [info] at scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202) > [info] at scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254) > [info] at scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254) > [info] at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222) > [info] at scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891) > [info] at scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891) > [info] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) > [info] at scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890) > [info] at scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110) > [info] at org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:31) > [info] at org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83) > [info] at org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83) > [info] at scala.Option.getOrElse(Option.scala:120) > [info] at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:83) > [info] at org.apache.spark.sql.cassandra.CassandraSQLContext.cassandraSql(CassandraSQLContext.scala:53) > [info] at org.apache.spark.sql.cassandra.CassandraSQLContext.sql(CassandraSQLContext.scala:56) > [info] at com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply$mcV$sp(CassandraSQLSpec.scala:169) > [info] at com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168) > [info] at com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168) > [info] at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22) > [info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85) > [info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) > [info] at org.scalatest.Transformer.apply(Transformer.scala:22) > [info] at org.scalatest.Transformer.apply(Transformer.scala:20) > [info] at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1647) > [info] at org.scalatest.Suite$class.withFixture(Suite.scala:1122) > [info] at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1683) > [info] at org.scalatest.FlatSpecLike$class.invokeWithFixture$1(FlatSpecLike.scala:1644) > [info] at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656) > [info] at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656) > [info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306) > [info] at org.scalatest.FlatSpecLike$class.runTest(FlatSpecLike.scala:1656) > [info] at org.scalatest.FlatSpec.runTest(FlatSpec.scala:1683) > [info] at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714) > [info] at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714) > [info] at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413) > [info] at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401) > [info] at scala.collection.immutable.List.foreach(List.scala:318) > [info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401) > [info] at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396) > [info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483) > [info] at org.scalatest.FlatSpecLike$class.runTests(FlatSpecLike.scala:1714) > [info] at org.scalatest.FlatSpec.runTests(FlatSpec.scala:1683) > [info] at org.scalatest.Suite$class.run(Suite.scala:1424) > [info] at org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:1683) > [info] at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1760) > [info] at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1760) > [info] at org.scalatest.SuperEngine.runImpl(Engine.scala:545) > [info] at org.scalatest.FlatSpecLike$class.run(FlatSpecLike.scala:1760) > [info] at org.scalatest.FlatSpec.run(FlatSpec.scala:1683) > [info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:466) > [info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:677) > [info] at sbt.ForkMain$Run$2.call(ForkMain.java:294) > [info] at sbt.ForkMain$Run$2.call(ForkMain.java:284) > [info] at java.util.concurrent.FutureTask.run(FutureTask.java:262) > [info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [info] at java.lang.Thread.run(Thread.java:745) > [info] - should allow to select rows with union distinct clause *** FAILED *** (46 milliseconds) > [info] java.lang.RuntimeException: [1.29] failure: ``UNION'' expected but `.' found > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org For additional commands, e-mail: issues-help@spark.apache.org