spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saiph Kappa <saiph.ka...@gmail.com>
Subject Re: Question about textFileStream
Date Wed, 12 Nov 2014 17:36:07 GMT
What if the window is of 5 seconds, and the file takes longer than 5
seconds to be completely scanned? It will still attempt to load the whole
file?

On Mon, Nov 10, 2014 at 6:24 PM, Soumitra Kumar <kumar.soumitra@gmail.com>
wrote:

> Entire file in a window.
>
> On Mon, Nov 10, 2014 at 9:20 AM, Saiph Kappa <saiph.kappa@gmail.com>
> wrote:
>
>> Hi,
>>
>> In my application I am doing something like this "new
>> StreamingContext(sparkConf, Seconds(10)).textFileStream("logs/")", and I
>> get some unknown exceptions when I copy a file with about 800 MB to that
>> folder ("logs/"). I have a single worker running with 512 MB of memory.
>>
>> Anyone can tell me if every 10 seconds spark reads parts of that big
>> file, or if it attempts to read the entire file in a single window? How
>> does it work?
>>
>> Thanks.
>>
>>
>

Mime
View raw message