[ https://issues.apache.org/jira/browse/SPARK-5358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14288430#comment-14288430
]
Matthew Whelan commented on SPARK-5358:
---------------------------------------
The PR for 2996 issue seems to cover SPARK-5358, though the bugs are slightly different.
> spark.files.userClassPathFirst doesn't work correctly
> -----------------------------------------------------
>
> Key: SPARK-5358
> URL: https://issues.apache.org/jira/browse/SPARK-5358
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.1.0, 1.2.0
> Reporter: Matthew Whelan
>
> org.apache.spark.executor.ChildExecutorURLClassLoader#findClass delegates to two different
classloaders: a parent-less URL classloader, and the parent classloader, as a fallback. The
delegation is broken such that calling loadClass twice in succession with the same parameters
will fail the second time.
> The delegation to the userClassLoader calls findClass directly, which bypasses the classloader's
cache. So userClassLoader will attempt to define the same class multiple times, throwing
LinkageErrors after the first time.
> The canonical way to change the default delegation scheme is to override loadClass, rather
than just findClass. It also might be sufficient to have userClassLoader.findClass call super.loadClass.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org
|