spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: CPU RAM
Date Wed, 17 Sep 2014 06:48:46 GMT
Ganglia does give you a cluster wide and per machine utilization of
resources, but i don't think it gives your per Spark Job. If you want to
build something from scratch then you can follow up like :

1. Login to the machine
2. Get the PIDs
3. For network IO per process, you can have a look at
http://nethogs.sourceforge.net/
4. You can make use of the information in /proc/[pid]/stat and /proc/stat
to estimate CPU usage and all


Similarly you can get any metric of process once you have the PID.


Thanks
Best Regards

On Wed, Sep 17, 2014 at 8:59 AM, VJ Shalish <vjshalish@gmail.com> wrote:

> Sorry for the confusion Team.
> My requirement is to measure the CPU utilisation, RAM usage, Network IO
> and other metrics of a SPARK JOB using Java program.
> Please help on the same.
>
> On Tue, Sep 16, 2014 at 11:23 PM, Amit <kumaramit01@gmail.com> wrote:
>
>> Not particularly related to Spark, but you can check out SIGAR API. It
>> let's you get CPU, Memory, Network, Filesystem and process based metrics.
>>
>> Amit
>> On Sep 16, 2014, at 20:14, VJ Shalish <vjshalish@gmail.com> wrote:
>>
>> > Hi
>> >
>> > I need to get the CPU utilisation, RAM usage, Network IO and other
>> metrics using Java program. Can anyone help me on this?
>> >
>> > Thanks
>> > Shalish.
>>
>
>

Mime
View raw message