spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthew Whelan (JIRA)" <>
Subject [jira] [Commented] (SPARK-5358) spark.files.userClassPathFirst doesn't work correctly
Date Thu, 22 Jan 2015 23:20:35 GMT


Matthew Whelan commented on SPARK-5358:

The PR for 2996 issue seems to cover SPARK-5358, though the bugs are slightly different.

> spark.files.userClassPathFirst doesn't work correctly
> -----------------------------------------------------
>                 Key: SPARK-5358
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.1.0, 1.2.0
>            Reporter: Matthew Whelan
> org.apache.spark.executor.ChildExecutorURLClassLoader#findClass delegates to two different
classloaders: a parent-less URL classloader, and the parent classloader, as a fallback.  The
delegation is broken such that calling loadClass twice in succession with the same parameters
will fail the second time.  
> The delegation to the userClassLoader calls findClass directly, which bypasses the classloader's
cache.  So userClassLoader will attempt to define the same class multiple times, throwing
LinkageErrors after the first time.
> The canonical way to change the default delegation scheme is to override loadClass, rather
than just findClass.  It also might be sufficient to have userClassLoader.findClass call super.loadClass.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message