spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Леонид Поляков <>
Subject Extra libs in executor classpath
Date Wed, 16 Mar 2016 11:22:14 GMT
Hello, guys!

I’ve been developing a kind of framework on top of spark, and my idea is to
bundle the framework jars and some extra configs with the spark and pass it
to other developers for their needs. So that devs can use this bundle and
run usual spark stuff but with extra flavor that framework will add.

I’m trying to figure out how to properly set up the driver/executor
classpath, so that framework classes are always loaded when you use the

I put framework libs in /lib folder right now, but will switch to something
more specific later. I’m putting next spark-defaults.conf into my bundle:

spark.executor.extraClassPath /home/user/Apps/spark-bundled/lib/*

spark.driver.extraClassPath lib/*

And this seem to work, but I want to get rid of the absolute path from
spark.executor.extraClassPath and use something relative, or spark home
somehow, since libs are right there under /lib

I’ve tried these settings for executor, and they do not work:

spark.executor.extraClassPath $SPARK_HOME/lib/*

spark.executor.extraClassPath lib/*

I’ve found out that work directory for started workers is like
$SPARK_HOME/work/app-20160316070310-0002/0, so this works:

spark.executor.extraClassPath ../../../lib/*

But looks cheaty and not stable.

Could you help me with this issue? Maybe there are some placeholders that I
can use in configs?

Let me know if you need any worker/master/driver logs

P.S. driver does not work if I am not in $SPARK_HOME when I execute
spark-submit, e.g. if I do

cd bin

./spark-submit …

Then driver classpath is relative to /bin and now lib/* or ./lib/* in
classpath does not work, so I need $SPARK_HOME for driver as well

Thanks, Leonid

View raw message