spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ayan guha <guha.a...@gmail.com>
Subject Re: spark dataframe rolling window for user define operation
Date Sat, 29 Oct 2016 14:18:09 GMT
Avg is an aggregation function. You need to write XYZ as user defined
aggregate function (UDAF).

On Sat, Oct 29, 2016 at 9:28 PM, Manjunath, Kiran <kimanjun@akamai.com>
wrote:

> Is there a way to get user defined operation to be used for rolling window
> operation?
>
>
>
> Like – Instead of
>
>
>
> val wSpec1 = Window.orderBy("c1").rowsBetween(-20, +20)
>
> var dfWithMovingAvg = df.withColumn( "Avg",avg(df("c2")).over(wSpec1))
>
>
>
> Something like
>
>
>
> val wSpec1 = Window.orderBy("c1").rowsBetween(-20, +20)
>
> var dfWithAlternate = df.withColumn( "alter",*XYZ*(df("c2")).over(wSpec1))
>
>
>
> Where XYZ function can be - +,-,+,- alternatively
>
>
>
>
>
> PS : I have posted the same question at http://stackoverflow.com/
> questions/40318010/spark-dataframe-rolling-window-user-define-operation
>
>
>
> Regards,
>
> Kiran
>



-- 
Best Regards,
Ayan Guha

Mime
View raw message