spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roberto Coluccio <roberto.coluc...@gmail.com>
Subject Re: Spark SQL weird exception after upgrading from 1.1.1 to 1.2.x
Date Wed, 18 Mar 2015 16:10:46 GMT
Hey Cheng, thank you so much for your suggestion, the problem was actually
a column/field called "timestamp" in one of the case classes!! Once I
changed its name everything worked out fine again. Let me say it was kinda
frustrating ...

Roberto

On Wed, Mar 18, 2015 at 4:07 PM, Roberto Coluccio <
roberto.coluccio@gmail.com> wrote:

> You know, I actually have one of the columns called "timestamp" ! This may
> really cause the problem reported in the bug you linked, I guess.
>
> On Wed, Mar 18, 2015 at 3:37 PM, Cheng Lian <lian.cs.zju@gmail.com> wrote:
>
>>  I suspect that you hit this bug
>> https://issues.apache.org/jira/browse/SPARK-6250, it depends on the
>> actual contents of your query.
>>
>> Yin had opened a PR for this, although not merged yet, it should be a
>> valid fix https://github.com/apache/spark/pull/5078
>>
>> This fix will be included in 1.3.1.
>>
>> Cheng
>>
>> On 3/18/15 10:04 PM, Roberto Coluccio wrote:
>>
>> Hi Cheng, thanks for your reply.
>>
>>  The query is something like:
>>
>>  SELECT * FROM (
>>>   SELECT m.column1, IF (d.columnA IS NOT null, d.columnA, m.column2),
>>> ..., m.columnN FROM tableD d RIGHT OUTER JOIN tableM m on m.column2 =
>>> d.columnA WHERE m.column2!=\"None\" AND d.columnA!=\"\"
>>>   UNION ALL
>>>   SELECT ... [another SELECT statement with different conditions but
>>> same tables]
>>>   UNION ALL
>>>   SELECT ... [another SELECT statement with different conditions but
>>> same tables]
>>> ) a
>>
>>
>>  I'm using just sqlContext, no hiveContext. Please, note once again that
>> this perfectly worked w/ Spark 1.1.x.
>>
>>  The tables, i.e. tableD and tableM are previously registered with the
>> RDD.registerTempTable method, where the input RDDs are actually a
>> RDD[MyCaseClassM/D], with MyCaseClassM and MyCaseClassD being simple
>> case classes with only (and less than 22) String fields.
>>
>>  Hope the situation is a bit more clear. Thanks anyone who will help me
>> out here.
>>
>>  Roberto
>>
>>
>>
>> On Wed, Mar 18, 2015 at 12:09 PM, Cheng Lian <lian.cs.zju@gmail.com>
>> wrote:
>>
>>>  Would you mind to provide the query? If it's confidential, could you
>>> please help constructing a query that reproduces this issue?
>>>
>>> Cheng
>>>
>>> On 3/18/15 6:03 PM, Roberto Coluccio wrote:
>>>
>>> Hi everybody,
>>>
>>>  When trying to upgrade from Spark 1.1.1 to Spark 1.2.x (tried both
>>> 1.2.0 and 1.2.1) I encounter a weird error never occurred before about
>>> which I'd kindly ask for any possible help.
>>>
>>>   In particular, all my Spark SQL queries fail with the following
>>> exception:
>>>
>>>  java.lang.RuntimeException: [1.218] failure: identifier expected
>>>>
>>>> [my query listed]
>>>>           ^
>>>>   at scala.sys.package$.error(package.scala:27)
>>>>   at
>>>> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:33)
>>>>   at
>>>> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
>>>>   at
>>>> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
>>>>   at
>>>> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:174)
>>>>   at
>>>> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:173)
>>>>   at
>>>> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
>>>>   at
>>>> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
>>>>   at
>>>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
>>>>   at
>>>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
>>>>   ...
>>>
>>>
>>>
>>>  The unit tests I've got for testing this stuff fail both if I
>>> build+test the project with Maven and if I run then as single ScalaTest
>>> files or test suites/packages.
>>>
>>>  When running my app as usual on EMR in YARN-cluster mode, I get the
>>> following:
>>>
>>>  15/03/17 11:32:14 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode:
15, (reason: User class threw exception: [1.218] failure: identifier expected
>>>
>>> SELECT * FROM ... (my query)
>>>                                                                             
                                                                                         
                                                  ^)
>>> Exception in thread "Driver" java.lang.RuntimeException: [1.218] failure: identifier
expected
>>>
>>> SELECT * FROM ... (my query)                                                
                                                                                         
                                                                                         
   ^
>>>     at scala.sys.package$.error(package.scala:27)
>>>     at org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:33)
>>>     at org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
>>>     at org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
>>>     at org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:174)
>>>     at org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:173)
>>>     at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
>>>     at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
>>>     at scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
>>>     at scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
>>>     at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
>>>     at scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
>>>     at scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
>>>     at scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
>>>     at scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
>>>     at scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
>>>     at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
>>>     at scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
>>>     at scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
>>>     at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
>>>     at scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
>>>     at scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
>>>     at org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:31)
>>>     at org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
>>>     at org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
>>>     at scala.Option.getOrElse(Option.scala:120)
>>>     at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:83)
>>>     at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:303)
>>>     at mycompany.mypackage.MyClassFunction.apply(MyClassFunction.scala:34)
>>>     at mycompany.mypackage.MyClass$.main(MyClass.scala:254)    at mycompany.mypackage.MyClass.main(MyClass.scala)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>     at java.lang.reflect.Method.invoke(Method.java:606)
>>>     at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:441)
>>> 15/03/17 11:32:14 INFO yarn.ApplicationMaster: Invoking sc stop from shutdown
hook
>>>
>>>  Any suggestions?
>>>
>>>  Thanks,
>>>
>>>  Roberto
>>>
>>>
>>>
>>
>>
>

Mime
View raw message