spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bin Chen <bc...@talend.com.INVALID>
Subject How to avoid spark executor fetching jars in Local mode
Date Fri, 11 Oct 2019 08:05:44 GMT
Hello everyone,

For spark local mode, I noticed the executor fetching jars from local, is there a way to avoid
these steps, as these jars already on the local filesystem, it takes seconds to fetching all
the jars, we want to reduce the time on bootstrap phase.

we are using spark-submit with `--conf spark.jars` to define these libs.

[2019-09-26 06:19:59,651] [INFO ] o.a.s.e.Executor - Fetching spark://10.2.1.5:43745/jars/xxx.jar
with timestamp 1569478797007
[2019-09-26 06:19:59,729] [INFO ] o.a.s.n.c.TransportClientFactory - Successfully created
connection to /10.2.1.5:43745 after 55 ms (0 ms spent in bootstraps)
[2019-09-26 06:19:59,754] [INFO ] o.a.s.u.Utils - Fetching spark://10.2.1.5:43745/jars/xxx.jar
to /tmp/spark-0365e48c-1747-4370-978f-7cd142ef0375/userFiles-3309dc5e-b6d0-4b76-a9aa-8e0a226ddab9/fetchFileTemp6346264674899227929.tmp
[2019-09-26 06:19:59,801] [INFO ] o.a.s.e.Executor - Adding file:/tmp/spark-0365e48c-1747-4370-978f-7cd142ef0375/userFiles-3309dc5e-b6d0-4b76-a9aa-8e0a226ddab9/xxx.jar
to class loader

Thanks

Chen Bin



As a recipient of an email from Talend, your contact personal data will be on our systems.
Please see our contacts privacy notice at Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>



Mime
View raw message