spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reynold Xin (JIRA)" <>
Subject [jira] [Resolved] (SPARK-13830) Fetch large directly result from executor is very slow
Date Sun, 01 May 2016 22:50:12 GMT


Reynold Xin resolved SPARK-13830.
       Resolution: Fixed
         Assignee: Davies Liu
    Fix Version/s: 2.0.0

> Fetch large directly result from executor is very slow
> ------------------------------------------------------
>                 Key: SPARK-13830
>                 URL:
>             Project: Spark
>          Issue Type: Task
>          Components: Spark Core
>            Reporter: Davies Liu
>            Assignee: Davies Liu
>             Fix For: 2.0.0
> Given two task with 100+M result on each, it take more than 50 seconds to fetch the results.
> The RPC may be not designed to handle large block, we should use block manager for that.
But currently this is based on spark.rpc.message.maxSize, which is usually very large (>
128M) for safe, it's too large for handling results.
> We also counting the time to fetch the direct result (also deserialize it) as schedule
delay, it also make sense to only fetch much smaller blocks via DirectResult.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message