spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yi Tian (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-6061) File source dstream can not include the old file which timestamp is before the system time
Date Thu, 05 Mar 2015 05:16:38 GMT

    [ https://issues.apache.org/jira/browse/SPARK-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348160#comment-14348160
] 

Yi Tian commented on SPARK-6061:
--------------------------------

In spark 1.2.1,  when you set the {{newFilesOnly}} to {{false}}, it means this {{FileInputDStream}}
would not only handle coming files, but also include files which came in the past 1 minute
(not all the old files). The length of time defined in {{FileInputDStream.MIN_REMEMBER_DURATION}}.
I think we should make this length of time configurable.


> File source dstream can not include the old file which timestamp is before the system
time
> ------------------------------------------------------------------------------------------
>
>                 Key: SPARK-6061
>                 URL: https://issues.apache.org/jira/browse/SPARK-6061
>             Project: Spark
>          Issue Type: Bug
>          Components: Streaming
>    Affects Versions: 1.2.1
>            Reporter: Jack Hu
>              Labels: FileSourceDStream, OlderFiles, Streaming
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> The file source dstream (StreamContext.fileStream) has a properties named "newFilesOnly"
to include the old files, it worked fine with 1.1.0, and broken at 1.2.1, the older files
always be ignored no mattern what value is set.  
> Here is the simple reproduce code:
> https://gist.github.com/jhu-chang/1ee5b0788c7479414eeb
> The reason is that: the "modTimeIgnoreThreshold" in FileInputDStream::findNewFiles is
set to a time closed to system time (Spark Streaming Clock time), so the files old than this
time are ignored. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message