spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bin Chen <>
Subject How to avoid spark executor fetching jars in Local mode
Date Fri, 11 Oct 2019 08:05:44 GMT
Hello everyone,

For spark local mode, I noticed the executor fetching jars from local, is there a way to avoid
these steps, as these jars already on the local filesystem, it takes seconds to fetching all
the jars, we want to reduce the time on bootstrap phase.

we are using spark-submit with `--conf spark.jars` to define these libs.

[2019-09-26 06:19:59,651] [INFO ] o.a.s.e.Executor - Fetching spark://
with timestamp 1569478797007
[2019-09-26 06:19:59,729] [INFO ] o.a.s.n.c.TransportClientFactory - Successfully created
connection to / after 55 ms (0 ms spent in bootstraps)
[2019-09-26 06:19:59,754] [INFO ] o.a.s.u.Utils - Fetching spark://
to /tmp/spark-0365e48c-1747-4370-978f-7cd142ef0375/userFiles-3309dc5e-b6d0-4b76-a9aa-8e0a226ddab9/fetchFileTemp6346264674899227929.tmp
[2019-09-26 06:19:59,801] [INFO ] o.a.s.e.Executor - Adding file:/tmp/spark-0365e48c-1747-4370-978f-7cd142ef0375/userFiles-3309dc5e-b6d0-4b76-a9aa-8e0a226ddab9/xxx.jar
to class loader


Chen Bin

As a recipient of an email from Talend, your contact personal data will be on our systems.
Please see our contacts privacy notice at Talend, Inc. <>

View raw message