spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Re: can HDFS be a streaming source like Kafka in Spark 2.2.0?
Date Tue, 16 Jan 2018 00:20:59 GMT
Hi,

I am not sure I understand. any examples ?

On Mon, Jan 15, 2018 at 3:45 PM, Gerard Maas <gerard.maas@gmail.com> wrote:

> Hi,
>
> You can monitor a filesystem directory as streaming source as long as the
> files placed there are atomically copied/moved into the directory.
> Updating the files is not supported.
>
> kr, Gerard.
>
> On Mon, Jan 15, 2018 at 11:41 PM, kant kodali <kanth909@gmail.com> wrote:
>
>> Hi All,
>>
>> I am wondering if HDFS can be a streaming source like Kafka in Spark
>> 2.2.0? For example can I have stream1 reading from Kafka and writing to
>> HDFS and stream2 to read from HDFS and write it back to Kakfa ? such that
>> stream2 will be pulling the latest updates written by stream1.
>>
>> Thanks!
>>
>
>

Mime
View raw message