spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Natu Lauchande <>
Subject Question Spark streaming - S3 textFileStream- How to get the current file name ?
Date Sat, 02 Apr 2016 04:17:55 GMT

I am using spark streamming and using the input strategy of  watching for
files in S3 directories.  Using the textFileStream method in the streamming

The filename contains relevant  for my pipeline manipulation i wonder if
there is a more robust way to get this name other than capturing RDD debug
information and parse the logs.


View raw message