spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "DEVAN M.S." <msdeva...@gmail.com>
Subject Re: IF statement doesn't work in Spark-SQL?
Date Tue, 20 Jan 2015 11:24:16 GMT
Add one more library

libraryDependencies += "org.apache.spark" % "spark-hive_2.10" % "1.2.0"


val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)

repalce sqlContext with hiveContext. Its working while using HiveContext
for me.



Devan M.S. | Research Associate | Cyber Security | AMRITA VISHWA
VIDYAPEETHAM | Amritapuri | Cell +919946535290 |


On Tue, Jan 20, 2015 at 4:45 PM, DEVAN M.S. <msdevanms@gmail.com> wrote:

> Which context are you using HiveContext or SQLContext ? Can you try with HiveContext
> ??
>
>
> Devan M.S. | Research Associate | Cyber Security | AMRITA VISHWA
> VIDYAPEETHAM | Amritapuri | Cell +919946535290 |
>
>
> On Tue, Jan 20, 2015 at 3:49 PM, Xuelin Cao <xuelincao2014@gmail.com>
> wrote:
>
>>
>> Hi, I'm using Spark 1.2
>>
>>
>> On Tue, Jan 20, 2015 at 5:59 PM, Wang, Daoyuan <daoyuan.wang@intel.com>
>> wrote:
>>
>>>  Hi Xuelin,
>>>
>>>
>>>
>>> What version of Spark are you using?
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Daoyuan
>>>
>>>
>>>
>>> *From:* Xuelin Cao [mailto:xuelincao2014@gmail.com]
>>> *Sent:* Tuesday, January 20, 2015 5:22 PM
>>> *To:* User
>>> *Subject:* IF statement doesn't work in Spark-SQL?
>>>
>>>
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>>       I'm trying to migrate some hive scripts to Spark-SQL. However, I
>>> found some statement is incompatible in Spark-sql.
>>>
>>>
>>>
>>>       Here is my SQL. And the same SQL works fine in HIVE environment.
>>>
>>>
>>>
>>>     SELECT
>>>
>>>       *if(ad_user_id>1000, 1000, ad_user_id) as user_id*
>>>
>>>     FROM
>>>
>>>       ad_search_keywords
>>>
>>>
>>>
>>>      What I found is, the parser reports error on the "*if*" statement:
>>>
>>>
>>>
>>> No function to evaluate expression. type: AttributeReference, tree:
>>> ad_user_id#4
>>>
>>>
>>>
>>>
>>>
>>>      Anyone have any idea about this?
>>>
>>>
>>>
>>>
>>>
>>
>>
>

Mime
View raw message