spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlos Fuertes (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-2016) rdd in-memory storage UI becomes unresponsive when the number of RDD partitions is large
Date Sat, 02 Aug 2014 01:04:08 GMT

    [ https://issues.apache.org/jira/browse/SPARK-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083263#comment-14083263
] 

Carlos Fuertes commented on SPARK-2016:
---------------------------------------

The real problem with the unresponsiveness of the browser is with the css style used for the
tables. I have updated in that same pull request the style of the table and that solves the
issue without having to modify how the data is currently presented (although we could still
add any other suggested options). See SPARK-2017

> rdd in-memory storage UI becomes unresponsive when the number of RDD partitions is large
> ----------------------------------------------------------------------------------------
>
>                 Key: SPARK-2016
>                 URL: https://issues.apache.org/jira/browse/SPARK-2016
>             Project: Spark
>          Issue Type: Sub-task
>            Reporter: Reynold Xin
>              Labels: starter
>
> Try run
> {code}
> sc.parallelize(1 to 100, 1000000).cache().count()
> {code}
> And open the storage UI for this RDD. It takes forever to load the page.
> When the number of partitions is very large, I think there are a few alternatives:
> 0. Only show the top 1000.
> 1. Pagination
> 2. Instead of grouping by RDD blocks, group by executors



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message