spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mayur Rustagi <mayur.rust...@gmail.com>
Subject Re: tracking resource usage for spark-shell commands
Date Wed, 26 Mar 2014 02:23:05 GMT
Time taken is shown in Shark shell web ui (hosted on 4040 port).
Also memory used is shown in terms of Storage of RDD, how much shuffle data
was written & read during the process is also highlighted thr.

Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>



On Tue, Mar 25, 2014 at 6:04 AM, Bharath Bhushan <manku.timma@outlook.com>wrote:

> Is there a way to see the resource usage of each spark-shell command — say
> time taken and memory used?
> I checked the WebUI of spark-shell and of the master and I don’t see any
> such breakdown. I see the time taken in the INFO logs but nothing about
> memory usage. It would also be nice to track the time taken in the
> spark-shell web UI.
>
> —
> Thanks

Mime
View raw message