spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: objectFile uses only java serializer?
Date Mon, 23 Mar 2015 20:31:27 GMT
bq. it uses Utils.deserialize, which is always using Java serialization.

I agree with your finding.

On Mon, Mar 23, 2015 at 1:14 PM, Koert Kuipers <koert@tresata.com> wrote:

> in the comments on SparkContext.objectFile it says:
> "It will also be pretty slow if you use the default serializer (Java
> serialization)"
>
> this suggests the spark.serializer is used, which means i can switch to
> the much faster kryo serializer. however when i look at the code it uses
> Utils.deserialize, which is always using Java serialization.
>
> did i get that right? and is this desired?
> it seems straightforward to switch objectFile to use the serializer as
> specified by spark.serializer (although it might being in new classloader
> issues).
>

Mime
View raw message