spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vamshi Talla <vamsh...@hotmail.com>
Subject Re: How to avoid duplicate column names after join with multiple conditions
Date Mon, 09 Jul 2018 02:39:43 GMT
Nirav,

Spark does not create a duplicate column when you use the below join expression,  as an array
of column(s) like below but that requires the column name to be same in both the data frames.

Example: df1.join(df2, [‘a’])

Thanks.
Vamshi Talla

On Jul 6, 2018, at 4:47 PM, Gokula Krishnan D <email2dgk@gmail.com<mailto:email2dgk@gmail.com>>
wrote:

Nirav,

withColumnRenamed() API might help but it does not different column and renames all the occurrences
of the given column. either use select() API and rename as you want.



Thanks & Regards,
Gokula Krishnan (Gokul)

On Mon, Jul 2, 2018 at 5:52 PM, Nirav Patel <npatel@xactlycorp.com<mailto:npatel@xactlycorp.com>>
wrote:
Expr is `df1(a) === df2(a) and df1(b) === df2(c)`

How to avoid duplicate column 'a' in result? I don't see any api that combines both. Rename
manually?



[What's New with Xactly]<https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.xactlycorp.com%2Femail-click%2F&data=02%7C01%7C%7C8ab8d95c23f44dfb156708d5e381c938%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636665068928877949&sdata=p4D%2FKz%2B%2Fd8wWFg9EGtNMRNcnYk5LlZmjQKx0TeWleDE%3D&reserved=0>

[https://www.xactlycorp.com/wp-content/uploads/2017/09/insta.png]<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.instagram.com%2Fxactlycorp%2F&data=02%7C01%7C%7C8ab8d95c23f44dfb156708d5e381c938%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636665068929034245&sdata=wtbLs3%2FABfsz8b1vN46EOcI22VZE1T5bhqOi9l1NFT0%3D&reserved=0>
 [https://www.xactlycorp.com/wp-content/uploads/2017/09/linkedin.png] <https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fxactly-corporation&data=02%7C01%7C%7C8ab8d95c23f44dfb156708d5e381c938%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636665068929034245&sdata=vyQkePM9Y3zG94CKUFJNtuAcEk6M60AtvhOjsHxBhbY%3D&reserved=0>
  [https://www.xactlycorp.com/wp-content/uploads/2017/09/twitter.png] <https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FXactly&data=02%7C01%7C%7C8ab8d95c23f44dfb156708d5e381c938%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636665068929034245&sdata=tRidhL1X4x4TPWdUHfH8%2Bcw8r7MT9jrRh1%2BJyU0LGCg%3D&reserved=0>
  [https://www.xactlycorp.com/wp-content/uploads/2017/09/facebook.png] <https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FXactlyCorp&data=02%7C01%7C%7C8ab8d95c23f44dfb156708d5e381c938%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636665068929034245&sdata=kh0aKmjvcG1ox5%2FMjdI5Ib%2FMvTu4xomFPLUcWDyBir8%3D&reserved=0>
  [https://www.xactlycorp.com/wp-content/uploads/2017/09/youtube.png] <https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.youtube.com%2Fxactlycorporation&data=02%7C01%7C%7C8ab8d95c23f44dfb156708d5e381c938%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636665068929034245&sdata=sicYYnUCmLBbOnUpu2v3Mp7btkt%2FEGWVMHHC%2BqFIenE%3D&reserved=0>


Mime
View raw message