spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <hol...@pigscanfly.ca>
Subject Re: Access to Applications metrics
Date Wed, 06 Dec 2017 01:18:09 GMT
I've done a SparkListener to record metrics for validation (it's a bit out
of date). Are you just looking to have graphing/alerting set up on the
Spark metrics?

On Tue, Dec 5, 2017 at 1:53 PM, Thakrar, Jayesh <
jthakrar@conversantmedia.com> wrote:

> You can also get the metrics from the Spark application events log file.
>
>
>
> See https://www.slideshare.net/JayeshThakrar/apache-
> bigdata2017sparkprofiling
>
>
>
>
>
> *From: *"Qiao, Richard" <Richard.Qiao@capitalone.com>
> *Date: *Monday, December 4, 2017 at 6:09 PM
> *To: *Nick Dimiduk <ndimiduk@gmail.com>, "user@spark.apache.org" <
> user@spark.apache.org>
> *Subject: *Re: Access to Applications metrics
>
>
>
> It works to collect Job level, through Jolokia java agent.
>
>
>
> Best Regards
>
> Richard
>
>
>
>
>
> *From: *Nick Dimiduk <ndimiduk@gmail.com>
> *Date: *Monday, December 4, 2017 at 6:53 PM
> *To: *"user@spark.apache.org" <user@spark.apache.org>
> *Subject: *Re: Access to Applications metrics
>
>
>
> Bump.
>
>
>
> On Wed, Nov 15, 2017 at 2:28 PM, Nick Dimiduk <ndimiduk@gmail.com> wrote:
>
> Hello,
>
>
>
> I'm wondering if it's possible to get access to the detailed
> job/stage/task level metrics via the metrics system (JMX, Graphite, &c).
> I've enabled the wildcard sink and I do not see them. It seems these values
> are only available over http/json and to SparkListener instances, is this
> the case? Has anyone worked on a SparkListener that would bridge data from
> one to the other?
>
>
>
> Thanks,
>
> Nick
>
>
>
>
> ------------------------------
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>



-- 
Twitter: https://twitter.com/holdenkarau

Mime
View raw message