spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-17671) Spark 2.0 history server summary page is slow even set spark.history.ui.maxApplications
Date Tue, 27 Sep 2016 00:18:20 GMT

    [ https://issues.apache.org/jira/browse/SPARK-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524587#comment-15524587
] 

Apache Spark commented on SPARK-17671:
--------------------------------------

User 'wgtmac' has created a pull request for this issue:
https://github.com/apache/spark/pull/15248

> Spark 2.0 history server summary page is slow even set spark.history.ui.maxApplications
> ---------------------------------------------------------------------------------------
>
>                 Key: SPARK-17671
>                 URL: https://issues.apache.org/jira/browse/SPARK-17671
>             Project: Spark
>          Issue Type: Bug
>          Components: Web UI
>    Affects Versions: 2.0.0
>            Reporter: Gang Wu
>
> This is a subsequent task of [SPARK-17243|https://issues.apache.org/jira/browse/SPARK-17243].
After the fix of SPARK-17243 (limit the number of applications in the JSON string transferred
from history server backend to web UI frontend), the history server does display the target
number of history summaries. 
> However, when there are more than 10k application history, it still gets slower and slower.
The problem is in the following code:
> {code:title=ApplicationListResource.scala|borderStyle=solid}
> @Produces(Array(MediaType.APPLICATION_JSON))
> private[v1] class ApplicationListResource(uiRoot: UIRoot) {
>   @GET
>   def appList(
>       @QueryParam("status") status: JList[ApplicationStatus],
>       @DefaultValue("2010-01-01") @QueryParam("minDate") minDate: SimpleDateParam,
>       @DefaultValue("3000-01-01") @QueryParam("maxDate") maxDate: SimpleDateParam,
>       @QueryParam("limit") limit: Integer)
>   : Iterator[ApplicationInfo] = {
>     // although there is a limit operation in the end
>     // the following line still does a transformation for all history 
>     // in the list
>     val allApps = uiRoot.getApplicationInfoList
>     
>     // ...
>     // irrelevant code is omitted 
>     // ...
>     if (limit != null) {
>       appList.take(limit)
>     } else {
>       appList
>     }
>   }
> }
> {code}
> What the code **uiRoot.getApplicationInfoList** does is to transform every application
history from class ApplicationHistoryInfo to class ApplicationInfo. So if there are 10k applications,
10k transformations will be done even we have limited 5000 jobs here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message