spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <van...@cloudera.com>
Subject Re: SparkLauncer 2.0.1 version working incosistently in yarn-client mode
Date Mon, 07 Nov 2016 17:46:01 GMT
On Sat, Nov 5, 2016 at 2:54 AM, Elkhan Dadashov <elkhan8502@gmail.com> wrote:
> while (appHandle.getState() == null || !appHandle.getState().isFinal()) {
>     if (appHandle.getState() != null) {
>         log.info("while: Spark job state is : " + appHandle.getState());
>         if (appHandle.getAppId() != null) {
>             log.info("\t App id: " + appHandle.getAppId() + "\tState: " +
> appHandle.getState());
>         }
>     }
> }

This is a ridiculously expensive busy loop, even more so if you
comment out the log lines. Use listeners, or at least sleep a little
bit every once in a while. You're probably starving other processes /
threads of cpu.

-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message