spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Zhou <zhou10...@gmail.com>
Subject Re: how to refresh the loaded non-streaming dataframe for each steaming batch ?
Date Fri, 06 Sep 2019 21:18:28 GMT
Not yet. Learning spark

On Fri, Sep 6, 2019 at 2:17 PM Shyam P <shyamabigdata@gmail.com> wrote:

> cool ,but did you find a way or anyhelp or clue ?
>
> On Fri, Sep 6, 2019 at 11:40 PM David Zhou <zhou10097@gmail.com> wrote:
>
>> I have the same question with yours
>>
>> On Thu, Sep 5, 2019 at 9:18 PM Shyam P <shyamabigdata@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am using spark-sql-2.4.1v to streaming in my PoC.
>>>
>>> how to refresh the loaded dataframe from hdfs/cassandra table every time
>>> new batch of stream processed ? What is the practice followed in general to
>>> handle this kind of scenario?
>>>
>>> Below is the SOF link for more details .
>>>
>>>
>>> https://stackoverflow.com/questions/57815645/how-to-refresh-the-contents-of-non-streaming-dataframe
>>>
>>> Thank you,
>>> Shyam
>>>
>>>
>>

Mime
View raw message