spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Krasheninnikov <>
Subject Re: Profiling a spark job
Date Mon, 11 Apr 2016 22:02:23 GMT
If you are profiling in standalone mode, I recommend you to try with Java
Mission Control.
You just need to start app with these params:

-XX:+UnlockCommercialFeatures -XX:+FlightRecorder$YOUR_PORT

And them connect with profiling agent, and perform recording. I have strong
opinion, that you have epoll wait time spent in one concrete thread, but
not in all spawned threads.

On Tue, Apr 5, 2016 at 1:34 PM, Dmitry Olshansky <>

> Hi list,
> I'm curious as to what are the best practices of profiling spark apps? So
> far I tried following this guide with hprof and/or yourkit but the profile
> looks strange:
>  55% of time spent in EPollWait. However I'm using standalone mode with
> local master without starting separate daemon (could it be that I should?)
> ---
> Dmitry Olshansky
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

View raw message