spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deepak Sharma <deepakmc...@gmail.com>
Subject Re: Spark 2.0 - Join statement compile error
Date Tue, 23 Aug 2016 05:02:55 GMT
Hi Subhajit
Try this in your join:
*val* *df** = **sales_demand**.**join**(**product_master**,**sales_demand**.$"INVENTORY_ITEM_ID"
=**== **product_master**.$"INVENTORY_ITEM_ID",**"inner"**)*

On Tue, Aug 23, 2016 at 2:30 AM, Subhajit Purkayastha <spurkaya@p3si.net>
wrote:

> *All,*
>
>
>
> *I have the following dataFrames and the temp table. *
>
>
>
> *I am trying to create a new DF , the following statement is not compiling*
>
>
>
> *val* *df** = **sales_demand**.**join**(**product_master**,(*
> *sales_demand**.INVENTORY_ITEM_ID**==**product_master*
> *.INVENTORY_ITEM_ID),**joinType**=**"inner"**)*
>
>
>
>
>
>
>
> *What am I doing wrong?*
>
>
>
> *==Code===*
>
>
>
> *var* sales_order_sql_stmt = s"""SELECT ORDER_NUMBER , INVENTORY_ITEM_ID,
> ORGANIZATION_ID,
>
>           from_unixtime(unix_timestamp(SCHEDULE_SHIP_DATE,'yyyy-MM-dd'),
> 'yyyy-MM-dd') AS schedule_date
>
>       FROM sales_order_demand
>
>       WHERE unix_timestamp(SCHEDULE_SHIP_DATE,'yyyy-MM-dd') >= $
> planning_start_date  limit 10"""
>
>
>
>     *val* sales_demand = spark.sql (sales_order_sql_stmt)
>
>
>
>     //print the data
>
>     *sales_demand**.**collect**()*.foreach { println }
>
>
>
>
>
>     *val* product_sql_stmt = "select SEGMENT1,INVENTORY_ITEM_ID,ORGANIZATION_ID
> from product limit 10"
>
>     *val* product_master = spark.sql (product_sql_stmt)
>
>
>
>     //print the data
>
>     *product_master**.**collect**()*.foreach { println }
>
>
>
>     *val* *df** = **sales_demand**.**join**(**product_master**,(*
> *sales_demand**.INVENTORY_ITEM_ID**==**product_master*
> *.INVENTORY_ITEM_ID),**joinType**=**"inner"**)*
>
>
>
>
>
>
>
>    spark.stop()
>



-- 
Thanks
Deepak
www.bigdatabig.com
www.keosha.net

Mime
View raw message