spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <...@preferred.jp>
Subject Re: Communication between Driver and Executors
Date Mon, 17 Nov 2014 02:37:49 GMT
Hi again,

On Mon, Nov 17, 2014 at 8:16 AM, Tobias Pfeiffer <tgp@preferred.jp> wrote:
>
> I have been trying to mis-use broadcast as in
> - create a class with a boolean var, set to true
> - query this boolean on the executors as a prerequisite to process the
> next item
> - when I want to shutdown, I set the boolean to false and unpersist the
> broadcast variable (which will trigger re-delivery).
> This is very dirty, but it works with a "local[*]" master. Unfortunately,
> when deployed on YARN, the new value will never arrive at my executors.
>

In fact, it seems as if "change mutable object (like mutable list) and
unpersist in order to trigger redeploy" only works locally. When running on
YARN, even after an unpersist, the value will always be identical to what I
shipped first. Now I wonder what unpersist actually does in that case. Must
I call unpersist from an executor or from the driver?

Thanks
Tobias

Mime
View raw message