spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Lyubimov <>
Subject Re: Kryo not default?
Date Mon, 12 May 2014 23:47:13 GMT
On Mon, May 12, 2014 at 2:47 PM, Anand Avati <> wrote:

> Hi,
> Can someone share the reason why Kryo serializer is not the default?

why should it be?

On top of it, the only way to serialize a closure into the backend (even
now) is java serialization (which means java serialization is required of
all closure attributes)

> Is
> there anything to be careful about (because of which it is not enabled by
> default)?

Yes. Kind of stems from above. There's still a number of api calls that use
closure attributes to serialize data to backend (see fold(), for example).
which means even if you enable kryo, some api still requires java
serialization of an attribute.

I fixed parallelize(), collect() and something else that i don't remember
already in that regard, but i think even up till now there's still a number
of apis lingering whose data parameters  wouldn't work with kryo.

> Thanks!

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message