As I understand Spark expects the jar files to be available on all nodes or if applicable on HDFS directoryPutting Spark Jar files on HDFS1)the : jar cv0f spark-libs.jar -C $SPARK_HOME/jars/ . ) a HDFS the jars the application hdfs dfs -mkdir /jars ) Upload HDFS: hdfs dfs -put spark-libs.jar /jars ) a cluster, increase the the Spark so that you reduce the amount times a NodeManager will a remote copy hdfs dfs -setrep -w hdfs:///jars/spark-libs.jar ( the amount replicas proportional the total NodeManagers) ) $SPARK_HOME/conf/spark-defaults.conf spark.yarn.archive hdfs:///rhes75: /jars/spark-libs.jar. Similar below spark.yarn.archive=hdfs://rhes75: /jars/spark-libs.jarEvery node of Spark needs to have the same $SPARK_HOME/conf/spark-defaults.conf fileHTH
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On Thu, 12 Nov 2020 at 16:35, Russell Spitzer <email@example.com> wrote:--driver-class-path does not move jars, so it is dependent on your Spark resource manager (master). It is interpreted literally so if your files do not exist in the location you provide relative where the driver is run, they will not be placed on the classpath.Since the driver is responsible for moving jars specified in --jars, you cannot use a jar specified by --jars to be in driver-class-path, since the driver is already started and it's classpath is already set before any jars are moved.Some distributions may change this behavior though, but this is the jist of it.On Thu, Nov 12, 2020 at 10:02 AM Dominique De Vito <firstname.lastname@example.org> wrote:Hi,
spark-submit --jars additional1.jar,additional2.jar \ --driver-class-path additional1.jar:additional2.jar \ --conf spark.executor.extraClassPath=additional1.jar:additional2.jar \ --class MyClass main-application.jar
spark-submit --jars /a/b/some1.jar,/a/b/c/some2.jar \ --driver-class-path XXX:YYY \ --conf spark.executor.extraClassPath=XXX:YYY \ --class MyClass main-application.jar
spark-submit --jars /a/b/some1.jar,/a/b/c/some2.jar \ --driver-class-path some1.jar:some2.jar \ --conf spark.executor.extraClassPath=some1.jar:some2.jar \ --class MyClass main-application.jar
spark-submit --jars /a/b/some1.jar,/a/b/c/some2.jar \ --driver-class-path ./some1.jar:./some2.jar \ --conf spark.executor.extraClassPath=./some1.jar:./some2.jar \ --class MyClass main-application.jar