spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Bozarth (JIRA)" <>
Subject [jira] [Commented] (SPARK-13269) Expose more executor stats in stable status API
Date Tue, 03 May 2016 22:28:12 GMT


Alex Bozarth commented on SPARK-13269:

Hey [~andrewor14], I was interested in this and took a look at the two examples you gave and
am a bit confused at what exactly you actually want. You can currently see the used memory,
used disk space, and active task count for each executor by calling /applications/[app-id]/executors
or (in code) getting the ExecutorSummary class for each executor and checking activeTask,
memoryUsed, and diskUsed. Are these numbers different from what you were interested in surfacing?
I was unsure of how those example related to JobProgressListener as well.

> Expose more executor stats in stable status API
> -----------------------------------------------
>                 Key: SPARK-13269
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>            Reporter: Andrew Or
> Currently the stable status API is quite limited; it exposes only a small subset of the
things exposed by JobProgressListener. It is useful for very high level querying but falls
short when the developer wants to build an application on top of Spark with more integration.
> In this issue I propose that we expose at least two things:
> - Which executors are running tasks, and
> - Which executors cached how much in memory and on disk
> The goal is not to expose exactly these two things, but to expose something that would
allow the developer to learn about them. These concepts are very much fundamental in Spark's
design so there's almost no chance that they will go away in the future.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message