spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy Ryza (JIRA)" <>
Subject [jira] [Resolved] (SPARK-1035) Use a single mechanism for distributing jars on Yarn
Date Mon, 07 Apr 2014 20:07:14 GMT


Sandy Ryza resolved SPARK-1035.

    Resolution: Won't Fix

When I originally filed this, I didn't realize that jars could be added at runtime.  In light
of this, I don't think we can do much better than the current state of things.

> Use a single mechanism for distributing jars on Yarn
> ----------------------------------------------------
>                 Key: SPARK-1035
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 0.9.0
>            Reporter: Sandy Pérez González
> When running Spark on Yarn, the app jar is distributed through a different mechanism
than additional added jars. The app jar gets to every worker node as a Yarn local resource.
Additional jars only get to the app master, and the app master serves them to workers with
the HTTP file server.   Strangeness comes when an application addJar's the app jar, which
is a natural thing to do in mesos or standalone mode, but in Yarn mode, will try to distribute
the same jar through a different mechanism.  Using the same mechanism for both would eliminate
this issue, as well as greatly simplify debugging ClassNotFoundExceptions in workers.

This message was sent by Atlassian JIRA

View raw message