spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shyam P <shyamabigd...@gmail.com>
Subject Any advice how to do this usecase in spark sql ?
Date Wed, 14 Aug 2019 03:08:05 GMT
Hi,
Any advice how to do this in spark sql ?

I have a scenario as below

dataframe1   = loaded from an HDFS Parquet file.

dataframe2 =   read from a Kafka Stream.

If column1 of dataframe1 value in columnX value of dataframe2 , then I need
then I need to replace column1 value of dataframe1.

Else add column1 value of dataframe1 to dataframe2 as a new record.


In a sense need to implement a look up dataframe which is refresh-able.
For more information please check

https://stackoverflow.com/questions/57479581/how-to-do-this-scenario-in-spark-streaming?noredirect=1#comment101437596_57479581


 Let me know if u need more info

Thanks

Mime
View raw message