spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gang Wu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-17672) Spark 2.0 history server web Ui takes too long for a single application
Date Mon, 26 Sep 2016 22:54:20 GMT
Gang Wu created SPARK-17672:
-------------------------------

             Summary: Spark 2.0 history server web Ui takes too long for a single application
                 Key: SPARK-17672
                 URL: https://issues.apache.org/jira/browse/SPARK-17672
             Project: Spark
          Issue Type: Bug
          Components: Web UI
    Affects Versions: 2.0.0
            Reporter: Gang Wu


When there are 10K application history in the history server back end, it can take a very
long time to even get a single application history page. After some investigation, I found
the root cause was the following piece of code: 

{code:title=OneApplicationResource.scala|borderStyle=solid}
@Produces(Array(MediaType.APPLICATION_JSON))
private[v1] class OneApplicationResource(uiRoot: UIRoot) {

  @GET
  def getApp(@PathParam("appId") appId: String): ApplicationInfo = {
    val apps = uiRoot.getApplicationInfoList.find { _.id == appId }
    apps.getOrElse(throw new NotFoundException("unknown app: " + appId))
  }

}
{code}

Although all application history infos are stored in a LinkedHashMap, here to code transforms
the map to an iterator and then uses the find() api which is O(n) instead of O(1) from a map.get()
operation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message