spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From maxdml <>
Subject Re: How does lineage get passed down in RDDs
Date Tue, 09 Jun 2015 02:11:06 GMT
If I read the code correctly, in RDD.scala, each rdd keeps track of it's own
dependencies, (from Dependency.scala), and has methods to access to it's
/ancestors/ dependencies, thus being able to recompute the lineage (see
getNarrowAncestors() or getDependencies() in some rdd like UnionRDD).

So it doesn't looks like an RDD knows the whole lineage graph without having
to compute it, nor does that an RDD gives more than it's own identity as a
parent to a child RDD.

As a new user I may be mistaken so any veteran confirmation would be
appreciated :)

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message