spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Divya Gehlot <>
Subject [Spark1.6]:compare rows and add new column based on lookup
Date Fri, 05 Aug 2016 02:16:44 GMT
I am working with Spark 1.6 with scala  and using Dataframe API .
I have a use case where I  need to compare two rows and add entry in the
new column based on the lookup table
for example :
My DF looks like :
col1            col2          newCol1
street1     person1
street2      person1         area1
street3     person1          area3
street5     person2
street6     person2          area5
street7     person4
street9     person4           area7

loop up table looks like
street1 -> street2 - area1
street2 -> street 3 - area3
street5 -> street6 - area5
street 7-> street 9 - area 7

if person moving from street 1 to street 2 then he is reaching area 1

Would really appreciate the help.


View raw message