spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jack Kolokasis <koloka...@ics.forth.gr>
Subject Re: How executor Understand which RDDs needed to be persist from the submitted Task
Date Thu, 09 Jan 2020 15:24:59 GMT
Thanks for the help. I read that driver only send a function (task) to 
executors and the executors apply this function to their local RDD 
partitions.

Iacovos

On 1/9/20 5:03 PM, Wenchen Fan wrote:
> RDD has a flag `storageLevel` which will be set by calling persist(). 
> RDD will be serialized and sent to executors for running tasks. So 
> executors just look at RDD.storageLevel and store output in its block 
> manager when needed.
>
> On Thu, Jan 9, 2020 at 5:53 PM Jack Kolokasis <kolokasis@ics.forth.gr 
> <mailto:kolokasis@ics.forth.gr>> wrote:
>
>     Hello all,
>
>     I want to find when a Task that is sended by Driver to executor
>     contains
>     a call to function persist(). I am trying to read the submitted
>     function
>     that driver send to executor but I could not find any call to
>     persist()
>     method. Do you know how executor understand which RDDs needed to
>     be persist?
>
>     Thanks,
>     Iacovos Kolokasis
>
>     ---------------------------------------------------------------------
>     To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>     <mailto:dev-unsubscribe@spark.apache.org>
>

Mime
View raw message