spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vadim Bichutskiy <vadim.bichuts...@gmail.com>
Subject Re: textFileStream Question
Date Sun, 17 May 2015 19:24:39 GMT
This is cool. Thanks Akhil.

ᐧ

On Sun, May 17, 2015 at 11:25 AM, Akhil Das <akhil@sigmoidanalytics.com>
wrote:

> With file timestamp, you can actually see the finding new files logic
> from here
> <https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala#L172>
>
> Thanks
> Best Regards
>
> On Fri, May 15, 2015 at 2:25 AM, Vadim Bichutskiy <
> vadim.bichutskiy@gmail.com> wrote:
>
>> How does textFileStream work behind the scenes? How does Spark Streaming
>> know what files are new and need to be processed? Is it based on time
>> stamp, file name?
>>
>> Thanks,
>> Vadim
>> ᐧ
>>
>
>

Mime
View raw message