spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitriy Selivanov (JIRA)" <>
Subject [jira] [Commented] (SPARK-12635) More efficient (column batch) serialization for Python/R
Date Tue, 12 Jan 2016 14:47:40 GMT


Dmitriy Selivanov commented on SPARK-12635:

First, thanks to all SparkR and Spark developers. 
I just start to evaluate SparkR. I tried it several times (since it was in AMPLab), but before
1.6 there were too many rough edges. So I used Scala API. For now I see two main limiting
issues (and they are interconnected):

1. Lack of UDF in R interface.  I saw SPARK-6817.
2. And I think more important - lack of fast serialization / deserialization. I believe it
is impossible to develop useful R UDF interface without fast  serialization / deserialization.

Consider following example. I have tiny cached spark DF with nrow=300k, ncol=25 and I want
to collect it to local R session:
df_local <- collect(df)

Resulting R data.frame is only ~ 70mb!!, but it takes **120sec** to collect it to R. (compared
to **7sec** of df.toPandas() in pyspark).
I made some profiling. Almost all time is spent at this calls collect -> callJStatic ->
invokeJava -> readObject. readObject make a lot of read* calls from [deserialize.R](

So for now it **much** faster to write spark data.frame to simple plain csv/json and then
read it R. I didn't read python serialization. Is it diffrent from R? Why so dramatic difference
between R and Python?

> More efficient (column batch) serialization for Python/R
> --------------------------------------------------------
>                 Key: SPARK-12635
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: PySpark, SparkR, SQL
>            Reporter: Reynold Xin
> Serialization between Scala / Python / R is pretty slow. Python and R both work pretty
well with column batch interface (e.g. numpy arrays). Technically we should be able to just
pass column batches around with minimal serialization (maybe even zero copy memory).
> Note that this depends on some internal refactoring to use a column batch interface in
Spark SQL.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message