spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pala M Muthaia <mchett...@rocketfuelinc.com>
Subject Re: Issues with constants in Spark HiveQL queries
Date Wed, 14 Jan 2015 20:02:34 GMT
conversion_action_id is a int. We also tried a string column predicate with
single quotes string value and hit the same error stack.

On Wed, Jan 14, 2015 at 7:11 AM, Yana Kadiyska <yana.kadiyska@gmail.com>
wrote:

> Just a guess but what is the type of conversion_aciton_id? I do queries
> over an epoch all the time with no issues(where epoch's type is bigint).
> You can see the source here
> https://github.com/apache/spark/blob/v1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala
--
> not sure what ASTNode type: 291 but it sounds like it's not considered
> numeric? If it's a string it should be conversion_aciton_id=*'*20141210*'
> *(single quotes around the string)
>
> On Tue, Jan 13, 2015 at 5:25 PM, Pala M Muthaia <
> mchettiar@rocketfuelinc.com> wrote:
>
>> Hi,
>>
>> We are testing Spark SQL-Hive QL, on Spark 1.2.0. We have run some simple
>> queries successfully, but we hit the following issue whenever we attempt to
>> use a constant in the query predicate.
>>
>> It seems like an issue with parsing constant.
>>
>> Query: SELECT user_id FROM actions where conversion_aciton_id=20141210
>>
>> Error:
>> scala.NotImplementedError: No parse rules for ASTNode type: 291, text:
>> 20141210 :
>> 20141210
>>
>> Any ideas? This seems very basic, so we may be missing something basic,
>> but i haven't figured out what it is.
>>
>> ---
>>
>> Full shell output below:
>>
>> scala> sqlContext.sql("SELECT user_id FROM actions where
>> conversion_aciton_id=20141210")
>> 15/01/13 16:55:54 INFO ParseDriver: Parsing command: SELECT user_id FROM
>> actions where conversion_aciton_id=20141210
>> 15/01/13 16:55:54 INFO ParseDriver: Parse Completed
>> 15/01/13 16:55:54 INFO ParseDriver: Parsing command: SELECT user_id FROM
>> actions where conversion_aciton_id=20141210
>> 15/01/13 16:55:54 INFO ParseDriver: Parse Completed
>> java.lang.RuntimeException:
>> Unsupported language features in query: SELECT user_id FROM actions where
>> conversion_aciton_id=20141210
>> TOK_QUERY
>>   TOK_FROM
>>     TOK_TABREF
>>       TOK_TABNAME
>>         actions
>>   TOK_INSERT
>>     TOK_DESTINATION
>>       TOK_DIR
>>         TOK_TMP_FILE
>>     TOK_SELECT
>>       TOK_SELEXPR
>>         TOK_TABLE_OR_COL
>>           user_id
>>     TOK_WHERE
>>       =
>>         TOK_TABLE_OR_COL
>>           conversion_aciton_id
>>         20141210
>>
>> scala.NotImplementedError: No parse rules for ASTNode type: 291, text:
>> 20141210 :
>> 20141210
>> " +
>>
>> org.apache.spark.sql.hive.HiveQl$.nodeToExpr(HiveQl.scala:1110)
>>
>> at scala.sys.package$.error(package.scala:27)
>> at org.apache.spark.sql.hive.HiveQl$.createPlan(HiveQl.scala:251)
>> at
>> org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:50)
>> at
>> org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:49)
>> at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
>> at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
>> at
>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
>> at
>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
>> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
>> at
>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
>> at
>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
>> at scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
>> at
>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
>> at
>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
>> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
>> at
>> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
>> at
>> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
>> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
>> at scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
>> at
>> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
>> at
>> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:31)
>> at org.apache.spark.sql.hive.HiveQl$$anonfun$3.apply(HiveQl.scala:133)
>> at org.apache.spark.sql.hive.HiveQl$$anonfun$3.apply(HiveQl.scala:133)
>> at
>> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:174)
>> at
>> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:173)
>> at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
>> at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
>> at
>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
>> at
>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
>> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
>> at
>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
>> at
>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
>> at scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
>> at
>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
>> at
>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
>> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
>> at
>> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
>> at
>> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
>> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
>> at scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
>> at
>> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
>> at
>> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:31)
>> at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:235)
>> at
>> org.apache.spark.sql.hive.HiveContext$$anonfun$sql$1.apply(HiveContext.scala:94)
>> at
>> org.apache.spark.sql.hive.HiveContext$$anonfun$sql$1.apply(HiveContext.scala:94)
>> at scala.Option.getOrElse(Option.scala:120)
>> at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:94)
>> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:15)
>> at $iwC$$iwC$$iwC.<init>(<console>:20)
>> at $iwC$$iwC.<init>(<console>:22)
>> at $iwC.<init>(<console>:24)
>> at <init>(<console>:26)
>> at .<init>(<console>:30)
>> at .<clinit>(<console>)
>> at .<init>(<console>:7)
>> at .<clinit>(<console>)
>> at $print(<console>)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:852)
>> at
>> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1125)
>> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:674)
>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:705)
>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:669)
>> at
>> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:828)
>> at
>> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:873)
>> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:785)
>> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:628)
>> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:636)
>> at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:641)
>> at
>> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:968)
>> at
>> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:916)
>> at
>> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:916)
>> at
>> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
>> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:916)
>> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1011)
>> at org.apache.spark.repl.Main$.main(Main.scala:31)
>> at org.apache.spark.repl.Main.main(Main.scala)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>
>

Mime
View raw message