spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yuri Oleynikov (יורי אולייניקוב <>
Subject Dynamic Spark metrics creation
Date Sat, 16 Jan 2021 13:20:42 GMT
Hi all,
I have a spark application with Arbitrary Stateful Aggregation implemented
with FlatMapGroupsWithStateFunction.

I want to make some statistics about incoming events inside
The statistics are made from some event property which on the one hand has
dynamic values but on the other hand - small finite set (thought unknown)
of values (e.g. country name).

So I thought to register dynamic metrics inside
FlatMapGroupsWithStateFunction but as far as I understand, this requires
accessing MetricsSystem via SparkEnv.get() which is unavailable from

Any thoughts/suggestions?

With best regards,

View raw message