spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeffrey Charles <jeffrey.char...@vidyard.com>
Subject Re: Dynamic allocation not releasing executors after unpersisting all cached data
Date Tue, 10 Jul 2018 16:54:18 GMT
Thanks for the suggestion. I gave it a try but the executor still isn't
being released several minutes after running that.

On Mon, Jul 9, 2018 at 3:51 PM Vadim Semenov <vadim@datadoghq.com> wrote:

> Try doing `unpersist(blocking=true)`
> On Mon, Jul 9, 2018 at 2:59 PM Jeffrey Charles
> <jeffrey.charles@vidyard.com> wrote:
> >
> > I'm persisting a dataframe in Zeppelin which has dynamic allocation
> enabled to get a sense of how much memory the dataframe takes up. After I
> note the size, I unpersist the dataframe. For some reason, Yarn is not
> releasing the executors that were added to Zeppelin. If I don't run the
> persist and unpersist steps, the executors that were added are removed
> about a minute after the paragraphs complete. Looking at the storage tab in
> the Spark UI for the Zeppelin job, I don't see anything cached.
> >
> > Is there any way to get Yarn to automatically remove executors after
> doing a persist followed by an unpersist if there is no activity on the
> executor within the configured dynamic allocation timeout (similar to how
> it works without a persist/unpersist cycle) without having to set
> spark.dynamicAllocation.cachedExecutorIdleTimeout? The main reason I'd like
> to avoid setting that configuration is I do not want to the executors being
> reclaimed if they do have cached data.
>
>
>
> --
> Sent from my iPhone
>

Mime
View raw message