spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Chen <tnac...@gmail.com>
Subject Re: [SPARK on MESOS] Avoid re-fetching Spark binary
Date Fri, 06 Jul 2018 09:18:11 GMT
Got it, then you can have an extracted Spark directory on each host on the
same location, and don’t specify SPARK_EXECUTOR_URI. Instead, set
spark.mesos.executor.home to that directory.

This should effectively do what you want, which avoids extracting and
fetching and just executed the command.

Tim
On Fri, Jul 6, 2018 at 5:57 PM Tien Dat <tphan.dat@gmail.com> wrote:

> Thank you for your answer.
>
> The think it I actually pointed to a local binary file. And Mesos locally
> copied the binary file to a specific folder in /var/lib/mesos/... and
> extract it to every time it launched an Spark executor. With the fetch
> cache, the copy time is reduced, but the reduction is not much since the
> file is stored at local any way.
> The process that takes more time is the extraction.
> Finally, since Mesos make a new folder for extracting the Spark binary each
> time a new Spark job runs, the disk usage increases gradually.
>
> Therefore, our expectation is to have Spark running on Mesos without this
> binary extraction, as well as without storing the same binary every time
> new
> Spark job runs.
>
> Does that make sense to you? And do you have any idea how to deal with
> this?
>
> Best
>
>
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Mime
View raw message