spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <>
Subject Re: Spark UI Source Code
Date Mon, 07 May 2018 16:20:54 GMT
On Mon, May 7, 2018 at 1:44 AM, Anshi Shrivastava
<> wrote:
> I've found a KVStore wrapper which stores all the metrics in a LevelDb
> store. This KVStore wrapper is available as a spark-dependency but we cannot
> access the metrics directly from spark since they are all private.

I'm not sure what it is you're trying to do exactly, but there's a
public REST API that exposes all the data Spark keeps about
applications. There's also a programmatic status tracker
(SparkContext.statusTracker) that's easier to use from within the
running Spark app, but has a lot less info.

> Can we use this store to store our own metrics?


> Also can we retrieve these metrics based on timestamp?

Only if the REST API has that feature, don't remember off the top of my head.


To unsubscribe e-mail:

View raw message