spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kappaganthu, Sivaram (ES)" <Sivaram.Kappagan...@ADP.com>
Subject Filename with DStreams
Date Wed, 05 Oct 2016 09:33:33 GMT

. Hi Forum,

I have seen the bow question in http://apache-spark-user-list.1001560.n3.nabble.com/Spark-streaming-textFileStream-fileStream-Get-file-name-td22692.html
.  Is there any solution discovered for this other than DebugString?? Please help.


Using spark streaming and listening to the files in HDFS using textFileStream/fileStream methods,
how do we get the fileNames which are read by these methods?

I used textFileStream which has file contents in JavaDStream and I got no success with fileStream
as it is throwing me a compilation error with spark version 1.3.1.

Can someone please tell me if we have an API function or any other way to get the file names
that these streaming methods read?

Thanks,
Sivaram

----------------------------------------------------------------------
This message and any attachments are intended only for the use of the addressee and may contain
information that is privileged and confidential. If the reader of the message is not the intended
recipient or an authorized representative of the intended recipient, you are hereby notified
that any dissemination of this communication is strictly prohibited. If you have received
this communication in error, notify the sender immediately by return email and delete the
message and any attachments from your system.

Mime
View raw message