spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Krasheninnikov <a.krasheninni...@corp.badoo.com>
Subject Re: Profiling a spark job
Date Mon, 11 Apr 2016 22:02:23 GMT
If you are profiling in standalone mode, I recommend you to try with Java
Mission Control.
You just need to start app with these params:

-XX:+UnlockCommercialFeatures -XX:+FlightRecorder
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=$YOUR_PORT
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false



And them connect with profiling agent, and perform recording. I have strong
opinion, that you have epoll wait time spent in one concrete thread, but
not in all spawned threads.

On Tue, Apr 5, 2016 at 1:34 PM, Dmitry Olshansky <dmitry.olsh@gmail.com>
wrote:

> Hi list,
>
> I'm curious as to what are the best practices of profiling spark apps? So
> far I tried following this guide with hprof and/or yourkit but the profile
> looks strange:
>
> https://cwiki.apache.org/confluence/display/SPARK/Profiling+Spark+Applications+Using+YourKit
>
>  55% of time spent in EPollWait. However I'm using standalone mode with
> local master without starting separate daemon (could it be that I should?)
>
> ---
> Dmitry Olshansky
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message