spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gang Wu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-17672) Spark 2.0 history server web Ui takes too long for a single application
Date Mon, 26 Sep 2016 22:55:20 GMT

    [ https://issues.apache.org/jira/browse/SPARK-17672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524424#comment-15524424
] 

Gang Wu commented on SPARK-17672:
---------------------------------

I'm working on a fix and will send a PR soon.

> Spark 2.0 history server web Ui takes too long for a single application
> -----------------------------------------------------------------------
>
>                 Key: SPARK-17672
>                 URL: https://issues.apache.org/jira/browse/SPARK-17672
>             Project: Spark
>          Issue Type: Bug
>          Components: Web UI
>    Affects Versions: 2.0.0
>            Reporter: Gang Wu
>
> When there are 10K application history in the history server back end, it can take a
very long time to even get a single application history page. After some investigation, I
found the root cause was the following piece of code: 
> {code:title=OneApplicationResource.scala|borderStyle=solid}
> @Produces(Array(MediaType.APPLICATION_JSON))
> private[v1] class OneApplicationResource(uiRoot: UIRoot) {
>   @GET
>   def getApp(@PathParam("appId") appId: String): ApplicationInfo = {
>     val apps = uiRoot.getApplicationInfoList.find { _.id == appId }
>     apps.getOrElse(throw new NotFoundException("unknown app: " + appId))
>   }
> }
> {code}
> Although all application history infos are stored in a LinkedHashMap, here to code transforms
the map to an iterator and then uses the find() api which is O( n) instead of O(1) from a
map.get() operation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message