spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Chen <>
Subject Re: [SPARK on MESOS] Avoid re-fetching Spark binary
Date Fri, 06 Jul 2018 09:18:11 GMT
Got it, then you can have an extracted Spark directory on each host on the
same location, and don’t specify SPARK_EXECUTOR_URI. Instead, set
spark.mesos.executor.home to that directory.

This should effectively do what you want, which avoids extracting and
fetching and just executed the command.

On Fri, Jul 6, 2018 at 5:57 PM Tien Dat <> wrote:

> Thank you for your answer.
> The think it I actually pointed to a local binary file. And Mesos locally
> copied the binary file to a specific folder in /var/lib/mesos/... and
> extract it to every time it launched an Spark executor. With the fetch
> cache, the copy time is reduced, but the reduction is not much since the
> file is stored at local any way.
> The process that takes more time is the extraction.
> Finally, since Mesos make a new folder for extracting the Spark binary each
> time a new Spark job runs, the disk usage increases gradually.
> Therefore, our expectation is to have Spark running on Mesos without this
> binary extraction, as well as without storing the same binary every time
> new
> Spark job runs.
> Does that make sense to you? And do you have any idea how to deal with
> this?
> Best
> --
> Sent from:
> ---------------------------------------------------------------------
> To unsubscribe e-mail:

View raw message