spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mathewwicks <>
Subject Do we support excluding the CURRENT ROW in PARTITION BY windowing functions?
Date Tue, 04 Apr 2017 02:21:34 GMT
Here is an example to illustrate my question. 

In this toy example, we are collecting a list of the other products that
each user has bought, and appending it as a new column. (Also note, that we
are filtering on some arbitrary column 'good_bad'.) 

I would like to know if we support NOT including the CURRENT ROW in the
OVER(PARTITION BY xxx) windowing function. 

For example, transaction 1 would have `other_purchases = [prod2, prod3]`
rather than `other_purchases = [prod1, prod2, prod3]`.

*------------------- Code Below -------------------*
df = spark.createDataFrame([ 
    (1, "user1", "prod1", "good"), 
    (2, "user1", "prod2", "good"), 
    (3, "user1", "prod3", "good"), 
    (4, "user2", "prod3", "bad"), 
    (5, "user2", "prod4", "good"), 
    (5, "user2", "prod5", "good")], 
    ("trans_id", "user_id", "prod_id", "good_bad") 

df = df.selectExpr( 
    "COLLECT_LIST(CASE WHEN good_bad == 'good' THEN prod_id END)
OVER(PARTITION BY user_id) AS other_purchases" 

Here is a stackoverflow link:

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe e-mail:

View raw message