spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: The equivalent for INSTR in Spark FP
Date Mon, 01 Aug 2016 21:13:20 GMT
Hi Mich,

There's no indexOf UDF -
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.functions$


Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Mon, Aug 1, 2016 at 7:24 PM, Mich Talebzadeh
<mich.talebzadeh@gmail.com> wrote:
> Hi,
>
> What is the equivalent of FP for the following window/analytic that works OK
> in Spark SQL
>
> This one using INSTR
>
> select
> substring(transactiondescription,1,INSTR(transactiondescription,'CD')-2),
>
>
> select distinct *
> from (
>       select
> substring(transactiondescription,1,INSTR(transactiondescription,'CD')-2),
>       SUM(debitamount) OVER (PARTITION BY
> substring(transactiondescription,1,INSTR(transactiondescription,'CD')-2)) AS
> spent
>       from accounts.ll_18740868 where transactiontype = 'DEB'
>      ) tmp
>
>
> I tried indexOf but it does not work!
>
> val wSpec2 =
> Window.partitionBy(substring(col("transactiondescription"),1,indexOf(col("transactiondescription"),"CD")))
> <console>:26: error: not found: value indexOf
>          val wSpec2 =
> Window.partitionBy(substring(col("transactiondescription"),1,indexOf(col("transactiondescription"),"CD")))
>
>
> Thanks
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> Disclaimer: Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed. The
> author will in no case be liable for any monetary damages arising from such
> loss, damage or destruction.
>
>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message