spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <>
Subject Re: textFileStream() issue?
Date Thu, 04 Dec 2014 01:07:51 GMT

On Wed, Dec 3, 2014 at 5:31 PM, Bahubali Jain <> wrote:
> I am trying to use textFileStream("some_hdfs_location") to pick new files
> from a HDFS location.I am seeing a pretty strange behavior though.
> textFileStream() is not detecting new files when I "move" them from a
> location with in hdfs to location at which textFileStream() is checking for
> new files.
> But when I copy files from a location in linux filesystem to hdfs then the
> textFileStream is detecting the new files.

Is it possible that the timestamp of the moved files is actually older than
the ones of previously processed files? I think only "new" files are picked
up. Try moving the file and set the timestamp to now() to see if it makes a


View raw message