spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <>
Subject Re: Profiling Spark tests with YourKit (or something else)
Date Mon, 14 Jul 2014 18:02:04 GMT
I haven't seen issues using the JVM's own tools (jstack, jmap, hprof and such), so maybe there's
a problem in YourKit or in your release of the JVM. Otherwise I'd suggest increasing the heap
size of the unit tests a bit (you can do this in the SBT build file). Maybe they are very
close to full and profiling pushes them over the edge.


On Jul 14, 2014, at 9:51 AM, Will Benton <> wrote:

> Hi all,
> I've been evaluating YourKit and would like to profile the heap and CPU usage of certain
tests from the Spark test suite.  In particular, I'm very interested in tracking heap usage
by allocation site.  Unfortunately, I get a lot of crashes running Spark tests with profiling
(and thus allocation-site tracking) enabled in YourKit; just using the sampler works fine,
but it appears that enabling the profiler breaks Utils.getCallSite.
> Is there a way to make this combination work?  If not, what are people using to understand
the memory and CPU behavior of Spark and Spark apps?
> thanks,
> wb

View raw message