spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuri Oleynikov (‫יורי אולייניקוב‬‎)" <>
Subject Re: Dynamic Spark metrics creation
Date Sun, 17 Jan 2021 14:15:39 GMT
Hey Jacek, I’ll clarify myself a bit:
As bottom line I need following metrics being reported by structured streaming:
Country-Poland: 23
Country-Brazil: 56

The country names are included in incoming events and unknown at very beginning/application

Thus registering accumulator and binding it to metric source at driver side on application
startup is impossible (unless you register with all possible country names - which is waste
of Spark memory, polluting metrics namespace with 99% of metrics having zero value, and wasting
the network bandwidth ).

Отправлено с iPhone

> 17 янв. 2021 г., в 15:56, Jacek Laskowski <> написал(а):
> Hey Yurii,
> > which is unavailable from executors.
> Register it on the driver and use accumulators on executors to update the values (on
the driver)?
> Pozdrawiam,
> Jacek Laskowski
> ----
> "The Internals Of" Online Books
> Follow me on
> ‪On Sat, Jan 16, 2021 at 2:21 PM ‫Yuri Oleynikov (יורי אולייניקוב‬‎
<> wrote:‬
>> Hi all, 
>> I have a spark application with Arbitrary Stateful Aggregation implemented with FlatMapGroupsWithStateFunction.
>> I want to make some statistics about incoming events inside FlatMapGroupsWithStateFunction.
>> The statistics are made from some event property which on the one hand has dynamic
values but on the other hand - small finite set (thought unknown) of values (e.g. country
>> So I thought to register dynamic metrics inside  FlatMapGroupsWithStateFunction but
as far as I understand, this requires accessing MetricsSystem via SparkEnv.get() which is
unavailable from executors.
>> Any thoughts/suggestions? 
>> With best regards,
>> Yurii

View raw message