spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dachuan Huang <huan...@cse.ohio-state.edu>
Subject Re: help me find the lineage graph code in spark
Date Sun, 20 Oct 2013 01:38:14 GMT
Thanks!


On Sat, Oct 19, 2013 at 9:11 PM, Mark Hamstra <mark@clearstorydata.com>wrote:

> What you are looking for is 'deps':
>
> abstract class RDD[T: ClassManifest](
>     @transient private var sc: SparkContext,
>     @transient private var deps: Seq[Dependency[_]]
>   ) extends Serializable with Logging {...}
>
>
>
> On Sat, Oct 19, 2013 at 5:53 PM, dachuan <hdc1112@gmail.com> wrote:
>
>> Hi, all,
>>
>> I am pretty new to spark internal world, and currently I am interested in
>> the fault tolerance in spark.
>>
>> a little surprised, i didn't find "lineage" keyword in directory
>> "incubator-spark/core/src/main/scala/org/apache/spark", I thought it must
>> be there because the nsdi 2012 paper mentioned it.
>>
>> [huangda@dachuan-desktop spark]$ find -type f | xargs grep -in "lineage"
>> [huangda@dachuan-desktop spark]$
>>
>> Could you please point me into the right source code file if you happen
>> to know?
>>
>> thank you,
>> dachuan.
>>
>>
>> --
>> Dachuan Huang
>> Cellphone: 614-390-7234
>> 2015 Neil Avenue
>> Ohio State University
>> Columbus, Ohio
>> U.S.A.
>> 43210
>>
>
>

Mime
View raw message