spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: [Spark Streaming] The FileInputDStream newFilesOnly=false does not work in 1.2 since
Date Wed, 21 Jan 2015 06:29:04 GMT
See also SPARK-3276 and SPARK-3553. Can you say more about the
problem? what are the file timestamps, what happens when you run, what
log messages if any are relevant. I do not expect there was any
intended behavior change.

On Wed, Jan 21, 2015 at 5:17 AM, Terry Hole <hujie.eagle@gmail.com> wrote:
> Hi,
>
> I am trying to move from 1.1 to 1.2 and found that the newFilesOnly=false
> (Intend to include old files) does not work anymore. It works great in 1.1,
> this should be introduced by the last change of this class.
>
>
>
> Does this flag behavior change or is it a regression?
>
> Issue should be caused by this code:
> From line 157 in FileInputDStream.scala
>     val modTimeIgnoreThreshold = math.max(
>         initialModTimeIgnoreThreshold,   // initial threshold based on
> newFilesOnly setting
>         currentTime - durationToRemember.milliseconds  // trailing end of
> the remember window
>       )
>
>
> Regards
>
> - Terry
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message