spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Commented] (SPARK-5358) spark.files.userClassPathFirst doesn't work correctly
Date Thu, 22 Jan 2015 13:16:34 GMT


Sean Owen commented on SPARK-5358:

I'm pretty sure this is the same issue as the old issue:
or at least it sounds very related.

Also see and

Are you in a position to propose a PR, or perhaps help close some of these as duplicates?
you might know better whether these sound like the same problem.

> spark.files.userClassPathFirst doesn't work correctly
> -----------------------------------------------------
>                 Key: SPARK-5358
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.1.0, 1.2.0
>            Reporter: Matthew Whelan
> org.apache.spark.executor.ChildExecutorURLClassLoader#findClass delegates to two different
classloaders: a parent-less URL classloader, and the parent classloader, as a fallback.  The
delegation is broken such that calling loadClass twice in succession with the same parameters
will fail the second time.  
> The delegation to the userClassLoader calls findClass directly, which bypasses the classloader's
cache.  So userClassLoader will attempt to define the same class multiple times, throwing
LinkageErrors after the first time.
> The canonical way to change the default delegation scheme is to override loadClass, rather
than just findClass.  It also might be sufficient to have userClassLoader.findClass call super.loadClass.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message