spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cheng Lian <lian.cs....@gmail.com>
Subject Re: Does SparkSQL not support nested IF(1=1, 1, IF(2=2, 2, 3)) statements?
Date Thu, 08 Jan 2015 13:48:44 GMT
The |+| operator only handles numeric data types, you may register you 
own concat function like this:

|sqlContext.registerFunction("concat", (s: String, t: String) => s + t)
sqlContext.sql("select concat('$', col1) from tbl")
|

Cheng

On 1/5/15 1:13 PM, RK wrote:

> The issue is happening when I try to concatenate column values in the 
> query like "col1+'
>
> +col3". For some reason, this issue is not manifesting itself when I 
> do a single IF query.
>
> Is there a concat function in SparkSQL? I can't find anything in the 
> documentation.
>
> Thanks,
> RK
>
>
>
> On Sunday, January 4, 2015 7:42 PM, RK <prk001@yahoo.com.INVALID> wrote:
>
>
> BTW, I am seeing this issue in Spark 1.1.1.
>
>
> On Sunday, January 4, 2015 7:29 PM, RK <prk001@yahoo.com.INVALID> wrote:
>
>
> When I use a single IF statement like "select IF(col1 != "", col1+'
>
> +col3, col2+'
>
> +col3) from my_table", it works fine.
>
> However, when I use a nested IF like "select IF(col1 != "", col1+'
>
> +col3, IF(col2 != "", col2+'
>
> +col3, '
>
> )) from my_table", I am getting the following exception.
>
> Exception in thread "main" 
> org.apache.spark.sql.catalyst.errors.package$TreeNodeException: 
> Unresolved attributes: if (NOT (col1#1 = )) (CAST((CAST(col1#1, 
> DoubleType) + CAST($, DoubleType)), DoubleType) + CAST(col3#3, 
> DoubleType)) else if (NOT (col2#2 = )) (CAST((CAST(col2#2, DoubleType) 
> + CAST($, DoubleType)), DoubleType) + CAST(col3#3, DoubleType)) else $ 
> AS c0#4, tree:
> Project [if (NOT (col1#1 = )) (CAST((CAST(col1#1, DoubleType) + 
> CAST($, DoubleType)), DoubleType) + CAST(col3#3, DoubleType)) else if 
> (NOT (col2#2 = )) (CAST((CAST(col2#2, DoubleType) + CAST($, 
> DoubleType)), DoubleType) + CAST(col3#3, DoubleType)) else $ AS c0#4]
>  Subquery my_table
>   SparkLogicalPlan (ExistingRdd [DB#0,col1#1,col2#2,col3#3], 
> MappedRDD[97] at getCallSite at DStream.scala:294)
>
> Does Spark SQL not support nested IF queries or is my query incorrect?
>
> Thanks,
> RK
>
>
>
>
‚Äč

Mime
View raw message