spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shyam P <>
Subject Any advice how to do this usecase in spark sql ?
Date Wed, 14 Aug 2019 03:08:05 GMT
Any advice how to do this in spark sql ?

I have a scenario as below

dataframe1   = loaded from an HDFS Parquet file.

dataframe2 =   read from a Kafka Stream.

If column1 of dataframe1 value in columnX value of dataframe2 , then I need
then I need to replace column1 value of dataframe1.

Else add column1 value of dataframe1 to dataframe2 as a new record.

In a sense need to implement a look up dataframe which is refresh-able.
For more information please check

 Let me know if u need more info


View raw message