spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yuri Oleynikov (יורי אולייניקוב <yur...@gmail.com>
Subject Dynamic Spark metrics creation
Date Sat, 16 Jan 2021 13:20:42 GMT
Hi all,
I have a spark application with Arbitrary Stateful Aggregation implemented
with FlatMapGroupsWithStateFunction.

I want to make some statistics about incoming events inside
FlatMapGroupsWithStateFunction.
The statistics are made from some event property which on the one hand has
dynamic values but on the other hand - small finite set (thought unknown)
of values (e.g. country name).

So I thought to register dynamic metrics inside
FlatMapGroupsWithStateFunction but as far as I understand, this requires
accessing MetricsSystem via SparkEnv.get() which is unavailable from
executors.

Any thoughts/suggestions?

With best regards,
Yurii

Mime
View raw message