spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From José Luis Pedrosa <>
Subject Spark 2.4.3 with hadoop 3.2 docker image.
Date Thu, 04 Jul 2019 17:13:08 GMT
Hi All

I'm trying to create docker images that can access azure services using
abfs hadoop driver, which is only available in haddop 3.2.

So I downloaded spark without Hadoop and generated spark images using the  itself.

In a new image using the resulting image as FROM, I've added hadoop 3.2
binary distro and following I've set:

export SPARK_DIST_CLASSPATH=$(/path/to/hadoop/bin/hadoop classpath)

Then when launching the jobs in K8s, it turns out, that the driver uses
internally spark-submit for the driver
it seems that launches with java directly for the executor
Result is that drivers can run correctly, but executors fails due to
missing sl4j class

Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(
at java.lang.Class.privateGetMethodRecursive(
at java.lang.Class.getMethod0(
at java.lang.Class.getMethod(
at sun.launcher.LauncherHelper.validateMainClass(
at sun.launcher.LauncherHelper.checkAndLoadMain(
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
at java.lang.ClassLoader.loadClass(
at sun.misc.Launcher$AppClassLoader.loadClass(
at java.lang.ClassLoader.loadClass(

If I'd add it manually to the class path, then another hadoop class would
be missing.

What is the right way to generate a docker image for spark 2.4 with a
custom hadoop distribution?

Thanks and regards

View raw message