spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-939) Allow user jars to take precedence over Spark jars, if desired
Date Sun, 30 Mar 2014 05:42:20 GMT

     [ https://issues.apache.org/jira/browse/SPARK-939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Patrick Wendell updated SPARK-939:
----------------------------------

    Reporter: Patrick Wendell  (was: NO NAME)

> Allow user jars to take precedence over Spark jars, if desired
> --------------------------------------------------------------
>
>                 Key: SPARK-939
>                 URL: https://issues.apache.org/jira/browse/SPARK-939
>             Project: Apache Spark
>          Issue Type: New Feature
>          Components: Spark Core
>            Reporter: Patrick Wendell
>            Assignee: holdenk
>            Priority: Blocker
>              Labels: starter
>             Fix For: 1.0.0
>
>
> Sometimes a user may want to include their own version of a jar that spark itself uses.
For example, if their code requires a newer version of that jar than Spark offers. It would
be good to have an option to give the users dependencies precedence over Spark. This options
should be disabled by default, since it could lead to some odd behavior (e.g. parts of Spark
not working). But I think we should have it.
> From an implementation perspective, this would require modifying the way we do class
loading inside of an Executor. The default behavior of the  URLClassLoader is to delegate
to it's parent first and, if that fails, to find a class locally. We want to have the opposite
behavior. This is sometimes referred to as "parent-last" (as opposed to "parent-first") class
loading precedence. There is an example of how to do this here:
> http://stackoverflow.com/questions/5445511/how-do-i-create-a-parent-last-child-first-classloader-in-java-or-how-to-overr
> We should write a similar class which can encapsulate a URL classloader and change the
delegation order. Or if possible, maybe we could find a more elegant way to do this. See relevant
discussion on the user list here:
> https://groups.google.com/forum/#!topic/spark-users/b278DW3e38g
> Also see the corresponding option in Hadoop:
> https://issues.apache.org/jira/browse/MAPREDUCE-4521
> Some other relevant Hadoop JIRA's:
> https://issues.apache.org/jira/browse/MAPREDUCE-1700
> https://issues.apache.org/jira/browse/MAPREDUCE-1938



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message