spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetić <otis.gospodne...@gmail.com>
Subject Re: Spark Executor Metrics
Date Tue, 16 Aug 2016 18:31:47 GMT
Hi Muhammad,

You should give people a bit more time to answer/help you (for free). :)

I don't have direct answer for you, but you can look at SPM for Spark
<https://sematext.com/blog/2014/10/07/apache-spark-monitoring/>, which has
all the instructions for getting all Spark metrics (Executors, etc.) into
SPM.  It doesn't involve sink.csv stuff.

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Tue, Aug 16, 2016 at 11:21 AM, Muhammad Haris <
muhammad.haris.makhtar@gmail.com> wrote:

> Still waiting for response, any clue/suggestions?
>
>
> On Tue, Aug 16, 2016 at 4:48 PM, Muhammad Haris <
> muhammad.haris.makhtar@gmail.com> wrote:
>
>> Hi,
>> I have been trying to collect driver, master, worker and executors
>> metrics using Spark 2.0 in standalone mode, here is what my metrics
>> configuration file looks like:
>>
>> *.sink.csv.class=org.apache.spark.metrics.sink.CsvSink
>> *.sink.csv.period=1
>> *.sink.csv.unit=seconds
>> *.sink.csv.directory=/root/metrics/
>> executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource
>> master.source.jvm.class=org.apache.spark.metrics.source.JvmSource
>> worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource
>> driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource
>>
>> Once application is complete, i can only see driver's metrics, have
>> checked directories on all the worker nodes as well.
>> Could anybody please help me what's i am doing wrong here.
>>
>>
>>
>> Regards
>>
>>
>>
>

Mime
View raw message