spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Nguyen <>
Subject Re: a question about lineage graphs in streaming
Date Sat, 02 Nov 2013 17:35:24 GMT
Dachuan, you may have correctly answered your own question. See Fig. 3 of
the same paper, where "infinity" occurs in the vertical direction.

Christopher T. Nguyen
Co-founder & CEO, Adatao <>

On Sat, Nov 2, 2013 at 7:51 AM, dachuan <> wrote:

> Hi, developers,
> I found this sentence hard to understand, it's from sosp'13 spark streaming
> paper:
> "Lineage cutoff: Because lineage graphs between RDDs
> in D-Streams can grow indefinitely, we modified the
> scheduler to forget lineage after an RDD has been checkpointed,
> so that its state does not grow arbitrarily."
> In my personal understanding, the length of DStream chain is fixed, so the
> RDDs these DStreams generate also have fixed length. Besides, the RDDs
> don't depend on the RDDs in the previous round. So why does it claim that
> lineage graph can grow indefinitely? when you say "grow indefinitely", do
> you refer to lineage graph's width or the number of lineage graphs?
> thanks,
> dachuan.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message