spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Appu K <>
Subject Re: Spark structured streaming: periodically refresh static data frame
Date Wed, 14 Feb 2018 10:57:37 GMT
More specifically,

Quoting TD from the previous thread
"Any streaming query that joins a streaming dataframe with the view will
automatically start using the most updated data as soon as the view is

Wondering if I’m doing something wrong in

My streaming dataframe is not using the updated data, even though the view
is updated!

Thank you

On 14 February 2018 at 2:54:48 PM, Appu K ( wrote:


I had followed the instructions from the thread
trying to reload a static data frame periodically that gets joined to a
structured streaming query.

However, the streaming query results does not reflect the data from the
refreshed static data frame.

Code is here

I’m using spark 2.2.1 . Any pointers would be highly helpful

Thanks a lot


View raw message