Can you share your code ?


Devan M.S. | Research Associate | Cyber Security | AMRITA VISHWA VIDYAPEETHAM | Amritapuri | Cell +919946535290 | 


On Tue, Jan 20, 2015 at 5:03 PM, Xuelin Cao <xuelincao2014@gmail.com> wrote:

Hi, 

     Yes, this is what I'm doing. I'm using hiveContext.hql() to run my query.

      But, the problem still happens.



On Tue, Jan 20, 2015 at 7:24 PM, DEVAN M.S. <msdevanms@gmail.com> wrote:
Add one more library

libraryDependencies += "org.apache.spark" % "spark-hive_2.10" % "1.2.0"


val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)

repalce sqlContext with hiveContext. Its working while using HiveContext for me.



Devan M.S. | Research Associate | Cyber Security | AMRITA VISHWA VIDYAPEETHAM | Amritapuri | Cell +919946535290


On Tue, Jan 20, 2015 at 4:45 PM, DEVAN M.S. <msdevanms@gmail.com> wrote:
Which context are you using HiveContext or SQLContext ? Can you try with HiveContext ??


Devan M.S. | Research Associate | Cyber Security | AMRITA VISHWA VIDYAPEETHAM | Amritapuri | Cell +919946535290


On Tue, Jan 20, 2015 at 3:49 PM, Xuelin Cao <xuelincao2014@gmail.com> wrote:

Hi, I'm using Spark 1.2


On Tue, Jan 20, 2015 at 5:59 PM, Wang, Daoyuan <daoyuan.wang@intel.com> wrote:

Hi Xuelin,

 

What version of Spark are you using?

 

Thanks,

Daoyuan

 

From: Xuelin Cao [mailto:xuelincao2014@gmail.com]
Sent: Tuesday, January 20, 2015 5:22 PM
To: User
Subject: IF statement doesn't work in Spark-SQL?

 

 

Hi,

 

      I'm trying to migrate some hive scripts to Spark-SQL. However, I found some statement is incompatible in Spark-sql.

 

      Here is my SQL. And the same SQL works fine in HIVE environment.

 

    SELECT

      if(ad_user_id>1000, 1000, ad_user_id) as user_id

    FROM

      ad_search_keywords

 

     What I found is, the parser reports error on the "if" statement:

 

No function to evaluate expression. type: AttributeReference, tree: ad_user_id#4

 

     

     Anyone have any idea about this?