spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hemanth Gudela <hemanth.gud...@qvantel.com>
Subject Spark structured streaming: Is it possible to periodically refresh static data frame?
Date Thu, 20 Apr 2017 20:08:57 GMT
Hello,

I am working on a use case where there is a need to join streaming data frame with a static
data frame.
The streaming data frame continuously gets data from Kafka topics, whereas static data frame
fetches data from a database table.

However, as the underlying database table is getting updated often, I must somehow manage
to refresh my static data frame periodically to get the latest information from underlying
database table.

My questions:

1.       Is it possible to periodically refresh static data frame?

2.       If refreshing static data frame is not possible, is there a mechanism to automatically
stop & restarting spark structured streaming job, so that every time the job restarts,
the static data frame gets updated with latest information from underlying database table.

3.       If 1) and 2) are not possible, please suggest alternatives to achieve my requirement
described above.

Thanks,
Hemanth
Mime
View raw message