spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shengshanzhang <shengshanzh...@icloud.com>
Subject Re: Spark 2.0 - Join statement compile error
Date Tue, 30 Aug 2016 09:55:31 GMT
Hi ,try this way

val df = sales_demand.join(product_master,sales_demand("INVENTORY_ITEM_ID") === product_master("INVENTORY_ITEM_ID"),"inner")

> 在 2016年8月30日,下午5:52,Jacek Laskowski <jacek@japila.pl> 写道:
> 
> Hi Mich,
> 
> This is the first time I've been told about $ for string interpolation (as the function
not the placeholder). Thanks for letting me know about it!
> 
> What is often used is s"whatever you want to reference inside the string $-prefix unless
it is a complex expression" i.e.
> 
> scala> s"I'm using $spark in ${spark.version}"
> res0: String = I'm using org.apache.spark.sql.SparkSession@1fc1c7e in 2.1.0-SNAPSHOT
> 
> 
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/ <https://medium.com/@jaceklaskowski/>
> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark <http://bit.ly/mastering-apache-spark>
> Follow me at https://twitter.com/jaceklaskowski <https://twitter.com/jaceklaskowski>
> On Tue, Aug 30, 2016 at 10:27 AM, Mich Talebzadeh <mich.talebzadeh@gmail.com <mailto:mich.talebzadeh@gmail.com>>
wrote:
> Actually I doubled checked this ‘s’ String Interpolator
> 
> In Scala
> 
> scala> val chars =  "This is Scala"
> chars: String = This is Scala
> scala> println($"$chars")
> This is Scala
> 
> OK so far fine. In shell (ksh) can do
> 
> chars="This is Scala"
> print "$chars"
> This is Scala
> 
> In Shell
> 
> print "$charsand it is interesting"
>  it is interesting
> 
> Have a problem with interpretation!
> 
> This will work by using {} brackers
> 
> print "${chars} and it is interesting"
> This is Scala and it is interesting
> 
> 
> In Scala
> 
> println($"$charsand it is interesting")
> <console>:24: error: not found: value charsand
>        println($"$charsand it is interesting")
> 
> Likewise shell this will work by deploying the brackets {}
> println($"${chars} and it is interesting")
> This is Scala and it is interesting
> 
> So I presume it is best practice to use ${} in Scala like shell? Although in most cases
$VALUE should work.
> 
> 
> Cheers
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>  
> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
> 
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage
or destruction of data or any other property which may arise from relying on this email's
technical content is explicitly disclaimed. The author will in no case be liable for any monetary
damages arising from such loss, damage or destruction.
>  
> 
> On 28 August 2016 at 09:25, Mich Talebzadeh <mich.talebzadeh@gmail.com <mailto:mich.talebzadeh@gmail.com>>
wrote:
> Yes I realised that. Actually I thought it was s not $. it has been around in shell for
years say for actual values --> ${LOG_FILE}, for position 's/ etc
> 
> 
> cat ${LOG_FILE} | egrep -v 'rows affected|return status|&&&' | sed -e 's/^[
           ]*//g' -e 's/^        //g' -e '/^$/d' > temp.out
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>  
> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
> 
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage
or destruction of data or any other property which may arise from relying on this email's
technical content is explicitly disclaimed. The author will in no case be liable for any monetary
damages arising from such loss, damage or destruction.
>  
> 
> On 28 August 2016 at 08:47, Jacek Laskowski <jacek@japila.pl <mailto:jacek@japila.pl>>
wrote:
> Hi Mich,
> 
> This is Scala's string interpolation which allow for replacing $-prefixed expressions
with their values.
> 
> It's what cool kids use in Scala to do templating and concatenation 😁
> 
> Jacek
> 
> 
> On 23 Aug 2016 9:21 a.m., "Mich Talebzadeh" <mich.talebzadeh@gmail.com <mailto:mich.talebzadeh@gmail.com>>
wrote:
> What is   --> s below before the text of sql?
> 
> var sales_order_sql_stmt = s"""SELECT ORDER_NUMBER , INVENTORY_ITEM_ID, ORGANIZATION_ID,
> 
>           from_unixtime(unix_timestamp(SCHEDULE_SHIP_DATE,'yyyy-MM-dd'), 'yyyy-MM-dd')
AS schedule_date
> 
>       FROM sales_order_demand
> 
>       WHERE unix_timestamp(SCHEDULE_SHIP_DATE,'yyyy-MM-dd') >= $planning_start_date
 limit 10"""
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>  
> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
> 
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage
or destruction of data or any other property which may arise from relying on this email's
technical content is explicitly disclaimed. The author will in no case be liable for any monetary
damages arising from such loss, damage or destruction.
>  
> 
> On 23 August 2016 at 07:31, Deepak Sharma <deepakmca05@gmail.com <mailto:deepakmca05@gmail.com>>
wrote:
> 
> On Tue, Aug 23, 2016 at 10:32 AM, Deepak Sharma <deepakmca05@gmail.com <mailto:deepakmca05@gmail.com>>
wrote:
> val df = sales_demand.join(product_master,sales_demand.$"INVENTORY_ITEM_ID" === product_master.$"INVENTORY_ITEM_ID","inner")
> 
> Ignore the last statement.
> It should look something like this:
> val df = sales_demand.join(product_master,$"sales_demand.INVENTORY_ITEM_ID" === $"product_master.INVENTORY_ITEM_ID","inner")
> 
> 
> -- 
> Thanks
> Deepak
> www.bigdatabig.com <http://www.bigdatabig.com/>
> www.keosha.net <http://www.keosha.net/>
> 
> 
> 


Mime
View raw message