spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saiph Kappa <saiph.ka...@gmail.com>
Subject Question about textFileStream
Date Mon, 10 Nov 2014 17:20:23 GMT
Hi,

In my application I am doing something like this "new
StreamingContext(sparkConf, Seconds(10)).textFileStream("logs/")", and I
get some unknown exceptions when I copy a file with about 800 MB to that
folder ("logs/"). I have a single worker running with 512 MB of memory.

Anyone can tell me if every 10 seconds spark reads parts of that big file,
or if it attempts to read the entire file in a single window? How does it
work?

Thanks.

Mime
View raw message