spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <...@preferred.jp>
Subject Re: textFileStream() issue?
Date Thu, 04 Dec 2014 01:07:51 GMT
Hi,

On Wed, Dec 3, 2014 at 5:31 PM, Bahubali Jain <bahubali@gmail.com> wrote:
>
> I am trying to use textFileStream("some_hdfs_location") to pick new files
> from a HDFS location.I am seeing a pretty strange behavior though.
> textFileStream() is not detecting new files when I "move" them from a
> location with in hdfs to location at which textFileStream() is checking for
> new files.
> But when I copy files from a location in linux filesystem to hdfs then the
> textFileStream is detecting the new files.
>

Is it possible that the timestamp of the moved files is actually older than
the ones of previously processed files? I think only "new" files are picked
up. Try moving the file and set the timestamp to now() to see if it makes a
difference.

Tobias

Mime
View raw message