spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Davidson <ilike...@gmail.com>
Subject Re: Profiling Spark tests with YourKit (or something else)
Date Mon, 14 Jul 2014 22:21:10 GMT
Out of curiosity, what problems are you seeing with Utils.getCallSite?


On Mon, Jul 14, 2014 at 2:59 PM, Will Benton <willb@redhat.com> wrote:

> Thanks, Matei; I have also had some success with jmap and friends and will
> probably just stick with them!
>
>
> best,
> wb
>
>
> ----- Original Message -----
> > From: "Matei Zaharia" <matei.zaharia@gmail.com>
> > To: dev@spark.apache.org
> > Sent: Monday, July 14, 2014 1:02:04 PM
> > Subject: Re: Profiling Spark tests with YourKit (or something else)
> >
> > I haven't seen issues using the JVM's own tools (jstack, jmap, hprof and
> > such), so maybe there's a problem in YourKit or in your release of the
> JVM.
> > Otherwise I'd suggest increasing the heap size of the unit tests a bit
> (you
> > can do this in the SBT build file). Maybe they are very close to full and
> > profiling pushes them over the edge.
> >
> > Matei
> >
> > On Jul 14, 2014, at 9:51 AM, Will Benton <willb@redhat.com> wrote:
> >
> > > Hi all,
> > >
> > > I've been evaluating YourKit and would like to profile the heap and CPU
> > > usage of certain tests from the Spark test suite.  In particular, I'm
> very
> > > interested in tracking heap usage by allocation site.  Unfortunately, I
> > > get a lot of crashes running Spark tests with profiling (and thus
> > > allocation-site tracking) enabled in YourKit; just using the sampler
> works
> > > fine, but it appears that enabling the profiler breaks
> Utils.getCallSite.
> > >
> > > Is there a way to make this combination work?  If not, what are people
> > > using to understand the memory and CPU behavior of Spark and Spark
> apps?
> > >
> > >
> > > thanks,
> > > wb
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message