spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <>
Subject [jira] [Commented] (SPARK-10901) spark.yarn.user.classpath.first doesn't work
Date Thu, 01 Oct 2015 19:18:26 GMT


Thomas Graves commented on SPARK-10901:

[~vanzin] thanks for the suggestion.  Did you work on the new user class path stuff at all?
 Wondering if you might have ideas why that didn't work.  It looks like its loading both versions.

> spark.yarn.user.classpath.first doesn't work
> --------------------------------------------
>                 Key: SPARK-10901
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.5.1
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>            Priority: Critical
> spark.yarn.user.classpath.first doesn't properly add the app jar to the system class
path first.  It has some logic there that i believe works for local files but running on yarn
using distributed cache to distribute the app jar doesn't put __app__.jar into the classpath
at all.
> This is a break in backwards compatibility.
> Note that in this case the user is trying to use different version of kryo (which used
to work in spark 1.2) and the new configs for this: spark.{driver, executor}.userClassPathFirst
don't allow this as it errors out with:
> User class threw exception: java.lang.LinkageError: loader constraint violation: loader
(instance of org/apache/spark/util/ChildFirstURLClassLoader) previously initiated loading
for a different type with name "com/esotericsoftware/kryo/Kryo"

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message