spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vadim Vararu <vadim.var...@adswizz.com>
Subject Lifecycle of a map function
Date Tue, 07 Apr 2020 08:44:13 GMT
Hi all,

I'm trying to guess understand what is the lifecycle of a map function in spark/yarn context.
My understanding is that function is instantiated on the master and then passed to each executor
(serialized/deserialized).

What I'd like to confirm is that the function is initialized/loaded/deserialized once per
executor (JVM in yarn) and lives as long as executor lives and not once per task (logical
unit of work to do).

Could you please explain or, better, give some links to source code or documentation? I've
tried to take a look in Task.scala and ResultTask.scala but I'm not familiar with Scala and
didn't find where exactly is function lifecycle managed.


Thanks in advance,
Vadim.

Mime
View raw message