spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Hamstra <m...@clearstorydata.com>
Subject Re: help me find the lineage graph code in spark
Date Sun, 20 Oct 2013 01:11:12 GMT
What you are looking for is 'deps':

abstract class RDD[T: ClassManifest](
    @transient private var sc: SparkContext,
    @transient private var deps: Seq[Dependency[_]]
  ) extends Serializable with Logging {...}



On Sat, Oct 19, 2013 at 5:53 PM, dachuan <hdc1112@gmail.com> wrote:

> Hi, all,
>
> I am pretty new to spark internal world, and currently I am interested in
> the fault tolerance in spark.
>
> a little surprised, i didn't find "lineage" keyword in directory
> "incubator-spark/core/src/main/scala/org/apache/spark", I thought it must
> be there because the nsdi 2012 paper mentioned it.
>
> [huangda@dachuan-desktop spark]$ find -type f | xargs grep -in "lineage"
> [huangda@dachuan-desktop spark]$
>
> Could you please point me into the right source code file if you happen to
> know?
>
> thank you,
> dachuan.
>
>
> --
> Dachuan Huang
> Cellphone: 614-390-7234
> 2015 Neil Avenue
> Ohio State University
> Columbus, Ohio
> U.S.A.
> 43210
>

Mime
View raw message