spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Ash (JIRA)" <>
Subject [jira] [Updated] (SPARK-3166) Custom serialisers can't be shipped in application jars
Date Sun, 05 Oct 2014 23:06:33 GMT


Andrew Ash updated SPARK-3166:
    Target Version/s: 1.2.0

> Custom serialisers can't be shipped in application jars
> -------------------------------------------------------
>                 Key: SPARK-3166
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.0.2
>            Reporter: Graham Dennis
> Spark cannot currently use a custom serialiser that is shipped with the application jar.
Trying to do this causes a java.lang.ClassNotFoundException when trying to instantiate the
custom serialiser in the Executor processes. This occurs because Spark attempts to instantiate
the custom serialiser before the application jar has been shipped to the Executor process.
A reproduction of the problem is available here:
> I've verified this problem in Spark 1.0.2, and Spark master and 1.1 branches as of August
21, 2014.  This issue is related to SPARK-2878, and my fix for that issue (
also solves this.  My pull request was not merged because it adds the user jar to the Executor
processes' class path at launch time.  Such a significant change was thought by [~rxin] to
require more QA, and should be considered for inclusion in 1.2 at the earliest.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message