spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephen Haberman (JIRA)" <>
Subject [jira] [Created] (SPARK-4877) userClassPathFirst doesn't handle user classes inheriting from parent
Date Wed, 17 Dec 2014 20:19:13 GMT
Stephen Haberman created SPARK-4877:

             Summary: userClassPathFirst doesn't handle user classes inheriting from parent
                 Key: SPARK-4877
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 1.2.0
            Reporter: Stephen Haberman

We're trying out userClassPathFirst.

To do so, we make an uberjar that does not contain Spark or Scala classes (because we want
those to load from the parent classloader, otherwise we'll get errors like scala.Function0
!= scala.Function0 since they'd load from different class loaders).

(Tangentially, some isolation classloaders like Jetty whitelist certain packages, like spark/*
and scala/*, to only come from the parent classloader, so that technically if the user still
messes up and leaks the Scala/Spark jars into their uberjar, it won't blow up; this would
be a good enhancement, I think.)

Anyway, we have a custom Kryo registrar, which ships in our uberjar, but since it "extends
spark.KryoRegistrator", which is not in our uberjar, we get a ClassNotFoundException.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message