spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xuelin Cao <xuelincao2...@gmail.com>
Subject Re: IF statement doesn't work in Spark-SQL?
Date Tue, 20 Jan 2015 11:33:48 GMT
Hi,

     Yes, this is what I'm doing. I'm using hiveContext.hql() to run my
query.

      But, the problem still happens.



On Tue, Jan 20, 2015 at 7:24 PM, DEVAN M.S. <msdevanms@gmail.com> wrote:

> Add one more library
>
> libraryDependencies += "org.apache.spark" % "spark-hive_2.10" % "1.2.0"
>
>
> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>
> repalce sqlContext with hiveContext. Its working while using HiveContext
> for me.
>
>
>
> Devan M.S. | Research Associate | Cyber Security | AMRITA VISHWA
> VIDYAPEETHAM | Amritapuri | Cell +919946535290 |
>
>
> On Tue, Jan 20, 2015 at 4:45 PM, DEVAN M.S. <msdevanms@gmail.com> wrote:
>
>> Which context are you using HiveContext or SQLContext ? Can you try with HiveContext
>> ??
>>
>>
>> Devan M.S. | Research Associate | Cyber Security | AMRITA VISHWA
>> VIDYAPEETHAM | Amritapuri | Cell +919946535290 |
>>
>>
>> On Tue, Jan 20, 2015 at 3:49 PM, Xuelin Cao <xuelincao2014@gmail.com>
>> wrote:
>>
>>>
>>> Hi, I'm using Spark 1.2
>>>
>>>
>>> On Tue, Jan 20, 2015 at 5:59 PM, Wang, Daoyuan <daoyuan.wang@intel.com>
>>> wrote:
>>>
>>>>  Hi Xuelin,
>>>>
>>>>
>>>>
>>>> What version of Spark are you using?
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Daoyuan
>>>>
>>>>
>>>>
>>>> *From:* Xuelin Cao [mailto:xuelincao2014@gmail.com]
>>>> *Sent:* Tuesday, January 20, 2015 5:22 PM
>>>> *To:* User
>>>> *Subject:* IF statement doesn't work in Spark-SQL?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>>       I'm trying to migrate some hive scripts to Spark-SQL. However, I
>>>> found some statement is incompatible in Spark-sql.
>>>>
>>>>
>>>>
>>>>       Here is my SQL. And the same SQL works fine in HIVE environment.
>>>>
>>>>
>>>>
>>>>     SELECT
>>>>
>>>>       *if(ad_user_id>1000, 1000, ad_user_id) as user_id*
>>>>
>>>>     FROM
>>>>
>>>>       ad_search_keywords
>>>>
>>>>
>>>>
>>>>      What I found is, the parser reports error on the "*if*" statement:
>>>>
>>>>
>>>>
>>>> No function to evaluate expression. type: AttributeReference, tree:
>>>> ad_user_id#4
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>      Anyone have any idea about this?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>

Mime
View raw message