spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: Generating a DStream by existing textfiles
Date Sun, 30 Nov 2014 07:14:44 GMT
If you look at the api doc
<https://spark.apache.org/docs/1.1.0/api/scala/index.html#org.apache.spark.streaming.StreamingContext>,
you can see the fileStream has a boolean parameter( newFilesOnly), setting
it false would pick up the existing files it seems.

Thanks
Best Regards

On Sun, Nov 30, 2014 at 4:46 AM, yu <yuz1988@iastate.edu> wrote:

> Hello Everyone,
>
> I am learning spark streaming and hope to find a convenient way to generate
> data stream from textfiles for some simple experiments. After I've viewed
> the scaladoc of spark, I found the methods 'textFileStream' and
> 'fileStream'
> could only monitor new files coming in but not existing files. Is there any
> method I could directly use in spark? For example, I have text1 in a
> folder,
> how can I generate DStream containing the data from text1?
>
> Thanks
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Generating-a-DStream-by-existing-textfiles-tp20030.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message