Hi !


I’m facing a Classloader problem using Spark 1.5.1


I use javax.validation and hibernate validation annotations on some of my beans :




  private String attribute1 ;



  private String attribute2 ;


When Spark tries to unmarshall these beans (after a remote RDD), I get the ClassNotFoundException :

17/09/07 09:19:25 INFO storage.BlockManager: Found block rdd_8_1 remotely

17/09/07 09:19:25 ERROR executor.Executor: Exception in task 3.0 in stage 2.0 (TID 6)

java.lang.ClassNotFoundException: org.hibernate.validator.constraints.NotBlank

        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:348)

        at java.io.ObjectInputStream.resolveProxyClass(ObjectInputStream.java:700)

        at java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1566)

        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)

        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)



Indeed, it means that the annotation class is not found, because it is not in the classpath. Why ? I don’t know, because I make a uber JAR that contains this class. I suppose that at the time the job tries to unmarshall the RDD, the uber jar is not loaded.


So, I try to add the hibernate JAR to the class loader manually, using this spark-submit command :


spark-submit --queue default \

--class com.my.Launcher \

--deploy-mode cluster \

--master yarn-cluster \

--driver-java-options "-Dfile.encoding=UTF-8" \

--jars /home/user/hibernate-validator-5.2.2.Final.jar \

--driver-class-path /home/user/hibernate-validator-5.2.2.Final.jar \

--conf "spark.executor.extraClassPath=/home/user/hibernate-validator-5.2.2.Final.jar" \



Without effects. So, is there a way to add this class to the classloader ?


Thank you in advance.




Ce message et toutes les pieces jointes (ci-apres le "message")
sont confidentiels et susceptibles de contenir des informations
couvertes par le secret professionnel. Ce message est etabli
a l'intention exclusive de ses destinataires. Toute utilisation
ou diffusion non autorisee interdite.
Tout message electronique est susceptible d'alteration. La SOCIETE GENERALE
et ses filiales declinent toute responsabilite au titre de ce message
s'il a ete altere, deforme falsifie.


This message and any attachments (the "message") are confidential,
intended solely for the addresses, and may contain legally privileged
information. Any unauthorized use or dissemination is prohibited.
E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any
of its subsidiaries or affiliates shall be liable for the message
if altered, changed or falsified.