spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: Resource usage of a spark application
Date Thu, 21 May 2015 11:00:12 GMT
Yes Peter that's correct, you need to identify the processes and with that
you can pull the actual usage metrics.

Thanks
Best Regards

On Thu, May 21, 2015 at 2:52 PM, Peter Prettenhofer <
peter.prettenhofer@gmail.com> wrote:

> Thanks Akhil, Ryan!
>
> @Akhil: YARN can only tell me how much vcores my app has been granted but
> not actual cpu usage, right? Pulling mem/cpu usage from the OS means i need
> to map JVM executor processes to the context they belong to, right?
>
> @Ryan: what a great blog post -- this is super relevant for me to analyze
> the state of the cluster as a whole. However, it seems to me that those
> metrics are mostly reported globally and not per spark application.
>
> 2015-05-19 21:43 GMT+02:00 Ryan Williams <ryan.blake.williams@gmail.com>:
>
>> Hi Peter, a few months ago I was using MetricsSystem to export to
>> Graphite and then view in Grafana; relevant scripts and some
>> instructions are here
>> <https://github.com/hammerlab/grafana-spark-dashboards/> if you want to
>> take a look.
>>
>>
>> On Sun, May 17, 2015 at 8:48 AM Peter Prettenhofer <
>> peter.prettenhofer@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I'm looking for a way to measure the current memory / cpu usage of a
>>> spark application to provide users feedback how much resources are actually
>>> being used.
>>> It seems that the metric system provides this information to some
>>> extend. It logs metrics on application level (nr of cores granted) and on
>>> the JVM level (memory usage).
>>> Is this the recommended way to gather this kind of information? If so,
>>> how do i best map a spark application to the corresponding JVM processes?
>>>
>>> If not, should i rather request this information from the resource
>>> manager (e.g. Mesos/YARN)?
>>>
>>> thanks,
>>>  Peter
>>>
>>> --
>>> Peter Prettenhofer
>>>
>>
>
>
> --
> Peter Prettenhofer
>

Mime
View raw message