spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Felix Cheung <>
Subject Re: Custom metrics sink
Date Sat, 17 Mar 2018 00:17:03 GMT
There is a proposal to expose them. See SPARK-14151

From: Christopher Piggott <>
Sent: Friday, March 16, 2018 1:09:38 PM
Subject: Custom metrics sink

Just for fun, i want to make a stupid program that makes different frequency chimes as each
worker becomes active.  That way you can 'hear' what the cluster is doing and how it's distributing

I thought to do this I would make a custom Sink, but the Sink and everything else in org.apache.spark.metrics.sink
is private to spark.  What I was hoping to do was to just pick up the # of active workers
in semi real time (once a second?) and have them send a UDP message somewhere... then each
worker would be assigned to a different frequency chime.  It's just a toy, for fun.

How do you add a custom Sink when these classes don't seem to be exposed?


View raw message