spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Sukmanowsky <mike.sukmanow...@gmail.com>
Subject Re: Spark Metrics Framework?
Date Tue, 22 Mar 2016 19:13:08 GMT
The Source class is private
<https://github.com/apache/spark/blob/v1.4.1/core/src/main/scala/org/apache/spark/metrics/source/Source.scala#L22-L25>
to the spark package and any new Sources added to the metrics registry must
be of type Source
<https://github.com/apache/spark/blob/v1.4.1/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala#L144-L152>.
So unless I'm mistaken, we can't define a custom source. I linked to 1.4.1
code, but the same is true in 1.6.1.

On Mon, 21 Mar 2016 at 12:05 Silvio Fiorito <silvio.fiorito@granturing.com>
wrote:

> You could use the metric sources and sinks described here:
> http://spark.apache.org/docs/latest/monitoring.html#metrics
>
> If you want to push the metrics to another system you can define a custom
> sink. You can also extend the metrics by defining a custom source.
>
> From: Mike Sukmanowsky <mike.sukmanowsky@gmail.com>
> Date: Monday, March 21, 2016 at 11:54 AM
> To: "user@spark.apache.org" <user@spark.apache.org>
> Subject: Spark Metrics Framework?
>
> We make extensive use of the elasticsearch-hadoop library for
> Hadoop/Spark. In trying to troubleshoot our Spark applications, it'd be
> very handy to have access to some of the many metrics
> <https://www.elastic.co/guide/en/elasticsearch/hadoop/current/metrics.html>
> that the library makes available when running in map reduce mode. The library's
> author noted
> <https://discuss.elastic.co/t/access-es-hadoop-stats-from-spark/44913>
> that Spark doesn't offer any kind of a similar metrics API where by these
> metrics could be reported or aggregated on.
>
> Are there any plans to bring a metrics framework similar to Hadoop's
> Counter system to Spark or is there an alternative means for us to grab
> metrics exposed when using Hadoop APIs to load/save RDDs?
>
> Thanks,
> Mike
>

Mime
View raw message