spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Or (JIRA)" <>
Subject [jira] [Assigned] (SPARK-12414) Remove closure serializer
Date Tue, 22 Dec 2015 20:54:46 GMT


Andrew Or reassigned SPARK-12414:

    Assignee: Andrew Or

> Remove closure serializer
> -------------------------
>                 Key: SPARK-12414
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Spark Core
>    Affects Versions: 1.0.0
>            Reporter: Andrew Or
>            Assignee: Andrew Or
> There is a config `spark.closure.serializer` that accepts exactly one value: the java
serializer. This is because there are currently bugs in the Kryo serializer that make it not
a viable candidate. This was uncovered by an unsuccessful attempt to make it work: SPARK-7708.
> My high level point is that the Java serializer has worked well for at least 6 Spark
versions now, and it is an incredibly complicated task to get other serializers (not just
Kryo) to work with Spark's closures. IMO the effort is not worth it and we should just remove
this documentation and all the code associated with it.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message