spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From manish ranjan <cse1.man...@gmail.com>
Subject Re: Monitoring the User Metrics for a long running Spark Job
Date Tue, 06 Dec 2016 01:53:34 GMT
http://spark.apache.org/docs/latest/monitoring.html

You can even install tools like  dstat
<http://dag.wieers.com/home-made/dstat/>, iostat
<http://linux.die.net/man/1/iostat>, and iotop
<http://linux.die.net/man/1/iotop>, *collectd*  can provide fine-grained
profiling on individual nodes.

If you are using Mesos as Resource Manager , mesos exposes metrics as well
for the running job.

Manish

~Manish



On Mon, Dec 5, 2016 at 4:17 PM, Chawla,Sumit <sumitkchawla@gmail.com> wrote:

> Hi All
>
> I have a long running job which takes hours and hours to process data.
> How can i monitor the operational efficency of this job?  I am interested
> in something like Storm\Flink style User metrics/aggregators, which i can
> monitor while my job is running.  Using these metrics i want to monitor,
> per partition performance in processing items.  As of now, only way for me
> to get these metrics is when the job finishes.
>
> One possibility is that spark can flush the metrics to external system
> every few seconds, and thus use  an external system to monitor these
> metrics.  However, i wanted to see if the spark supports any such use case
> OOB.
>
>
> Regards
> Sumit Chawla
>
>

Mime
View raw message