spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jack Hu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-3276) Provide a API to specify whether the old files need to be ignored in file input text DStream
Date Thu, 28 Aug 2014 04:43:57 GMT
Jack Hu created SPARK-3276:
------------------------------

             Summary: Provide a API to specify whether the old files need to be ignored in
file input text DStream
                 Key: SPARK-3276
                 URL: https://issues.apache.org/jira/browse/SPARK-3276
             Project: Spark
          Issue Type: Bug
          Components: Streaming
    Affects Versions: 1.0.2
            Reporter: Jack Hu


Currently, only one API called textFileStream in StreamingContext to specify the text file
dstream, which ignores the old files always. On some times, the old files is still useful.
Need a API to let user choose whether the old files need to be ingored or not .

The API currently in StreamingContext:
def textFileStream(directory: String): DStream[String] = {
    fileStream[LongWritable, Text, TextInputFormat](directory).map(_._2.toString)
  }



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message