spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liang-Chi Hsieh <>
Subject Re: the uniqueSource in StreamExecution, where is it be changed please?
Date Sat, 05 Aug 2017 09:14:10 GMT

Not sure if you are looking for how the returned value of `getOffset`

I think it depends on how the actual `Source` classes implement it. For
example, in `FileStreamSource`, you can see `getOffset` is updated by
finding new files in the source.

Different source has different way to get its offset.

萝卜丝炒饭 wrote
> Hi all,
> These days I am learning the code about the StreamExecution.
> In the method constructNextBatch(about line 365), I found the value of
> latestOffsets changed but I can not find where the s.getOffset of
> uniqueSource is  changed.
> here is the code link:
> Would you like help understand it please?
> Thanks.
> Robin

Liang-Chi Hsieh | @viirya 
Spark Technology Center 
View this message in context:
Sent from the Apache Spark Developers List mailing list archive at

To unsubscribe e-mail:

View raw message