spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Natu Lauchande <nlaucha...@gmail.com>
Subject Question Spark streaming - S3 textFileStream- How to get the current file name ?
Date Sat, 02 Apr 2016 04:17:55 GMT
Hi,

I am using spark streamming and using the input strategy of  watching for
files in S3 directories.  Using the textFileStream method in the streamming
context.

The filename contains relevant  for my pipeline manipulation i wonder if
there is a more robust way to get this name other than capturing RDD debug
information and parse the logs.

Thanks,
Natu

Mime
View raw message