spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gourav Sengupta <gourav.sengu...@gmail.com>
Subject Re:
Date Mon, 11 Sep 2017 13:32:33 GMT
Hi,

Why SPARK 1.5? Let me guess, are you using JAVA as well?

The world has moved on mate.


Regards,
Gourav

On Fri, Sep 8, 2017 at 8:36 AM, PICARD Damien <damien.picard-ext@socgen.com>
wrote:

> Hi !
>
>
>
> I’m facing a Classloader problem using Spark 1.5.1
>
>
>
> I use javax.validation and hibernate validation annotations on some of my
> beans :
>
>
>
>   @NotBlank
>
>   @Valid
>
>   *private* String attribute1 ;
>
>
>
>   @Valid
>
>   *private* String attribute2 ;
>
>
>
> When Spark tries to unmarshall these beans (after a remote RDD), I get the ClassNotFoundException
> :
>
> 17/09/07 09:19:25 INFO storage.BlockManager: Found block rdd_8_1 remotely
>
> 17/09/07 09:19:25 ERROR executor.Executor: Exception in task 3.0 in stage
> 2.0 (TID 6)
>
> java.lang.ClassNotFoundException: org.hibernate.validator.
> constraints.NotBlank
>
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
>         at java.lang.Class.forName0(Native Method)
>
>         at java.lang.Class.forName(Class.java:348)
>
>         at java.io.ObjectInputStream.resolveProxyClass(
> ObjectInputStream.java:700)
>
>         at java.io.ObjectInputStream.readProxyDesc(
> ObjectInputStream.java:1566)
>
>         at java.io.ObjectInputStream.readClassDesc(
> ObjectInputStream.java:1518)
>
>         at java.io.ObjectInputStream.readOrdinaryObject(
> ObjectInputStream.java:1781)
>
>    …
>
>
>
> Indeed, it means that the annotation class is not found, because it is not
> in the classpath. Why ? I don’t know, because I make a uber JAR that
> contains this class. I suppose that at the time the job tries to unmarshall
> the RDD, the uber jar is not loaded.
>
>
>
> So, I try to add the hibernate JAR to the class loader manually, using
> this spark-submit command :
>
>
>
> spark-submit --queue default \
>
> --class com.my.Launcher \
>
> --deploy-mode cluster \
>
> --master yarn-cluster \
>
> --driver-java-options "-Dfile.encoding=UTF-8" \
>
> --jars /home/user/hibernate-validator-5.2.2.Final.jar \
>
> --driver-class-path /home/user/hibernate-validator-5.2.2.Final.jar \
>
> --conf "spark.executor.extraClassPath=/home/user/
> hibernate-validator-5.2.2.Final.jar" \
>
> /home/user/uberjar-job.jar
>
>
>
> Without effects. So, is there a way to add this class to the classloader ?
>
>
>
> Thank you in advance.
>
>
>
> Damien
>
> =========================================================
>
> Ce message et toutes les pieces jointes (ci-apres le "message")
> sont confidentiels et susceptibles de contenir des informations
> couvertes par le secret professionnel. Ce message est etabli
> a l'intention exclusive de ses destinataires. Toute utilisation
> ou diffusion non autorisee interdite.
> Tout message electronique est susceptible d'alteration. La SOCIETE GENERALE
> et ses filiales declinent toute responsabilite au titre de ce message
> s'il a ete altere, deforme falsifie.
>
> =========================================================
>
> This message and any attachments (the "message") are confidential,
> intended solely for the addresses, and may contain legally privileged
> information. Any unauthorized use or dissemination is prohibited.
> E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any
> of its subsidiaries or affiliates shall be liable for the message
> if altered, changed or falsified.
>
> =========================================================
>

Mime
View raw message