spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anton Kirillov <akiril...@mesosphere.com>
Subject [METRICS] Metrics names inconsistent between executions
Date Mon, 06 May 2019 20:28:44 GMT
Hi everyone!

We are currently working on building a unified monitoring/alerting solution
for Spark and would like to rely on Spark's own metrics to avoid divergence
from the upstream. One of the challenges is to support metrics coming from
multiple Spark applications running on a cluster: scheduled jobs,
long-running streaming applications etc.

Original problem:
Spark assigns metrics names using *spark.app.id <http://spark.app.id>*
and *spark.executor.id
<http://spark.executor.id>* as a part of them. Thus the number of metrics
is continuously growing because those IDs are unique between executions
whereas the metrics themselves report the same thing. Another issue which
arises here is how to use constantly changing metric names in dashboards.

For example, *jvm_heap_used* reported by all Spark instances (components):
- <spark.app.id>_driver_jvm_heap_used (Driver)
- <spark.app.id>_<spark.executor.id>_jvm_heap_used (Executors)

While *spark.app.id <http://spark.app.id>* can be overridden with
*spark.metrics.namespace*, there's no such an option for *spark.executor.id
<http://spark.executor.id>* which makes it impossible to build a reusable
dashboard because (given the uniqueness of IDs) differently named metrics
are emitted for each execution.

One of the possible solutions would be to make executor metrics names
follow the driver's metrics name pattern, e.g.:
- <spark.app.id>_driver_jvm_heap_used (Driver)
- <spark.app.id>_executor_jvm_heap_used (Executors)

and distinguish executors based on tags (tags should be configured in
metric reporters in this case). Not sure if this could potentially break
Driver UI though.

I'd really appreciate any feedback on this issue and would be happy to
create a Jira issue/PR if this change looks sane for the community.

Thanks in advance.

-- 
*Anton Kirillov*
Senior Software Engineer, Mesosphere

Mime
View raw message