spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthew Whelan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-5358) spark.files.userClassPathFirst doesn't work correctly
Date Thu, 22 Jan 2015 23:20:35 GMT

    [ https://issues.apache.org/jira/browse/SPARK-5358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14288430#comment-14288430
] 

Matthew Whelan commented on SPARK-5358:
---------------------------------------

The PR for 2996 issue seems to cover SPARK-5358, though the bugs are slightly different.

> spark.files.userClassPathFirst doesn't work correctly
> -----------------------------------------------------
>
>                 Key: SPARK-5358
>                 URL: https://issues.apache.org/jira/browse/SPARK-5358
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.1.0, 1.2.0
>            Reporter: Matthew Whelan
>
> org.apache.spark.executor.ChildExecutorURLClassLoader#findClass delegates to two different
classloaders: a parent-less URL classloader, and the parent classloader, as a fallback.  The
delegation is broken such that calling loadClass twice in succession with the same parameters
will fail the second time.  
> The delegation to the userClassLoader calls findClass directly, which bypasses the classloader's
cache.  So userClassLoader will attempt to define the same class multiple times, throwing
LinkageErrors after the first time.
> The canonical way to change the default delegation scheme is to override loadClass, rather
than just findClass.  It also might be sufficient to have userClassLoader.findClass call super.loadClass.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message