spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <tathagata.das1...@gmail.com>
Subject Re: MEMORY_ONLY_SER question
Date Tue, 04 Nov 2014 21:02:41 GMT
It it deserialized in a streaming manner as the iterator moves over the
partition. This is a functionality of core Spark, and Spark Streaming just
uses it as is.
What do you want to customize it to?

On Tue, Nov 4, 2014 at 9:22 AM, Mohit Jaggi <mohitjaggi@gmail.com> wrote:

> Folks,
> If I have an RDD persisted in MEMORY_ONLY_SER mode and then it is needed
> for a transformation/action later, is the whole partition of the RDD
> deserialized into Java objects first before my transform/action code works
> on it? Or is it deserialized in a streaming manner as the iterator moves
> over the partition? Is this behavior customizable? I generally use the Kryo
> serializer.
>
> Mohit.
>

Mime
View raw message