spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: reuse the Spark SQL internal metrics
Date Tue, 30 Aug 2016 21:25:41 GMT
Hi,

If the stats are in web UI, they should be flying over the wire and so
you can catch the events by implementing SparkListener [1] -- a
developer API for custom Spark listeners. That's how web UI gets the
data and History Server. I think the stats are sent as accumulator
updates in onExecutorMetricsUpdate.

[1] http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.scheduler.SparkListener

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Tue, Aug 30, 2016 at 11:17 PM, Ai Deng <wx.london.cun@gmail.com> wrote:
> Hi there,
>
> I think the metrics inside of the different SparkPlan (like "numOutputRows"
> in FilterExec) are useful to build any Dev dashboard or business monitoring.
> Are there a easy way or exist solution to expose and persist these metrics
> out of Spark UI (ex: send to Graphite)? Currently they are all "private"
> inside Spark library.
>
> And the main benefit is you can get these metrics for "free" without change
> your Spark application.
>
> Thanks and regards,
>
> Ai
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/reuse-the-Spark-SQL-internal-metrics-tp27626.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message