spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mania Abdi <abdi...@husky.neu.edu>
Subject Spark DAG scheduler
Date Thu, 16 Apr 2020 22:16:23 GMT
Hello everyone,

I am implementing a caching mechanism for analytic workloads running on top
of Spark and I need to retrieve the Spark DAG right after it is generated
and the DAG scheduler. I would appreciate it if you could give me some
hints or reference me to some documents about where the DAG is generated
and inputs assigned to it. I found the DAG Scheduler class
<https://github.com/apache/spark/blob/55dea9be62019d64d5d76619e1551956c8bb64d0/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala>
but I am not sure if it is a good starting point.

Regards
Mania

Mime
View raw message