spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "DEVAN M.S." <msdeva...@gmail.com>
Subject Re: IF statement doesn't work in Spark-SQL?
Date Tue, 20 Jan 2015 12:02:29 GMT
Can you share your code ?


Devan M.S. | Research Associate | Cyber Security | AMRITA VISHWA
VIDYAPEETHAM | Amritapuri | Cell +919946535290 |


On Tue, Jan 20, 2015 at 5:03 PM, Xuelin Cao <xuelincao2014@gmail.com> wrote:

>
> Hi,
>
>      Yes, this is what I'm doing. I'm using hiveContext.hql() to run my
> query.
>
>       But, the problem still happens.
>
>
>
> On Tue, Jan 20, 2015 at 7:24 PM, DEVAN M.S. <msdevanms@gmail.com> wrote:
>
>> Add one more library
>>
>> libraryDependencies += "org.apache.spark" % "spark-hive_2.10" % "1.2.0"
>>
>>
>> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>>
>> repalce sqlContext with hiveContext. Its working while using HiveContext
>> for me.
>>
>>
>>
>> Devan M.S. | Research Associate | Cyber Security | AMRITA VISHWA
>> VIDYAPEETHAM | Amritapuri | Cell +919946535290 |
>>
>>
>> On Tue, Jan 20, 2015 at 4:45 PM, DEVAN M.S. <msdevanms@gmail.com> wrote:
>>
>>> Which context are you using HiveContext or SQLContext ? Can you try
>>> with HiveContext ??
>>>
>>>
>>> Devan M.S. | Research Associate | Cyber Security | AMRITA VISHWA
>>> VIDYAPEETHAM | Amritapuri | Cell +919946535290 |
>>>
>>>
>>> On Tue, Jan 20, 2015 at 3:49 PM, Xuelin Cao <xuelincao2014@gmail.com>
>>> wrote:
>>>
>>>>
>>>> Hi, I'm using Spark 1.2
>>>>
>>>>
>>>> On Tue, Jan 20, 2015 at 5:59 PM, Wang, Daoyuan <daoyuan.wang@intel.com>
>>>> wrote:
>>>>
>>>>>  Hi Xuelin,
>>>>>
>>>>>
>>>>>
>>>>> What version of Spark are you using?
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Daoyuan
>>>>>
>>>>>
>>>>>
>>>>> *From:* Xuelin Cao [mailto:xuelincao2014@gmail.com]
>>>>> *Sent:* Tuesday, January 20, 2015 5:22 PM
>>>>> *To:* User
>>>>> *Subject:* IF statement doesn't work in Spark-SQL?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>>       I'm trying to migrate some hive scripts to Spark-SQL. However,
I
>>>>> found some statement is incompatible in Spark-sql.
>>>>>
>>>>>
>>>>>
>>>>>       Here is my SQL. And the same SQL works fine in HIVE environment.
>>>>>
>>>>>
>>>>>
>>>>>     SELECT
>>>>>
>>>>>       *if(ad_user_id>1000, 1000, ad_user_id) as user_id*
>>>>>
>>>>>     FROM
>>>>>
>>>>>       ad_search_keywords
>>>>>
>>>>>
>>>>>
>>>>>      What I found is, the parser reports error on the "*if*"
>>>>> statement:
>>>>>
>>>>>
>>>>>
>>>>> No function to evaluate expression. type: AttributeReference, tree:
>>>>> ad_user_id#4
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>      Anyone have any idea about this?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message