spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tien Dat <>
Subject Re: [SPARK on MESOS] Avoid re-fetching Spark binary
Date Fri, 06 Jul 2018 08:57:49 GMT
Thank you for your answer.

The think it I actually pointed to a local binary file. And Mesos locally
copied the binary file to a specific folder in /var/lib/mesos/... and
extract it to every time it launched an Spark executor. With the fetch
cache, the copy time is reduced, but the reduction is not much since the
file is stored at local any way.
The process that takes more time is the extraction.
Finally, since Mesos make a new folder for extracting the Spark binary each
time a new Spark job runs, the disk usage increases gradually.

Therefore, our expectation is to have Spark running on Mesos without this
binary extraction, as well as without storing the same binary every time new
Spark job runs.

Does that make sense to you? And do you have any idea how to deal with this?


Sent from:

To unsubscribe e-mail:

View raw message