spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manjunath, Kiran" <kiman...@akamai.com>
Subject spark dataframe rolling window for user define operation
Date Sat, 29 Oct 2016 10:28:01 GMT
Is there a way to get user defined operation to be used for rolling window operation?

Like – Instead of

val wSpec1 = Window.orderBy("c1").rowsBetween(-20, +20)
var dfWithMovingAvg = df.withColumn( "Avg",avg(df("c2")).over(wSpec1))

Something like

val wSpec1 = Window.orderBy("c1").rowsBetween(-20, +20)
var dfWithAlternate = df.withColumn( "alter",XYZ(df("c2")).over(wSpec1))

Where XYZ function can be - +,-,+,- alternatively


PS : I have posted the same question at http://stackoverflow.com/questions/40318010/spark-dataframe-rolling-window-user-define-operation

Regards,
Kiran
Mime
View raw message