spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elkhan Dadashov <elkhan8...@gmail.com>
Subject Re: SparkLauncer 2.0.1 version working incosistently in yarn-client mode
Date Thu, 10 Nov 2016 08:31:23 GMT
Thanks Marcelo.

I changed the code using CountDownLatch, and it works as expected.

...final CountDownLatch countDownLatch = new CountDownLatch(1);
SparkAppListener sparkAppListener = new SparkAppListener(countDownLatch);
SparkAppHandle appHandle =
sparkLauncher.startApplication(sparkAppListener);Thread
sparkAppListenerThread = new Thread(sparkAppListener);
sparkAppListenerThread.start();long timeout = 120;
countDownLatch.await(timeout, TimeUnit.SECONDS);
...

    private static class SparkAppListener implements
SparkAppHandle.Listener, Runnable {
        private static final Log log =
LogFactory.getLog(SparkAppListener.class);
        private final CountDownLatch countDownLatch;
        public SparkAppListener(CountDownLatch countDownLatch) {
            this.countDownLatch = countDownLatch;
        }
        @Override
        public void stateChanged(SparkAppHandle handle) {
            String sparkAppId = handle.getAppId();
            State appState = handle.getState();
            if (sparkAppId != null) {
                log.info("Spark job with app id: " + sparkAppId + ",\t
State changed to: " + appState + " - "
                        + SPARK_STATE_MSG.get(appState));
            } else {
                log.info("Spark job's state changed to: " + appState +
" - " + SPARK_STATE_MSG.get(appState));
            }
            if (appState != null && appState.isFinal()) {
                countDownLatch.countDown();
            }
        }
        @Override
        public void infoChanged(SparkAppHandle handle) {}
        @Override
        public void run() {}
    }


On Mon, Nov 7, 2016 at 9:46 AM Marcelo Vanzin <vanzin@cloudera.com> wrote:

> On Sat, Nov 5, 2016 at 2:54 AM, Elkhan Dadashov <elkhan8502@gmail.com>
> wrote:
> > while (appHandle.getState() == null || !appHandle.getState().isFinal()) {
> >     if (appHandle.getState() != null) {
> >         log.info("while: Spark job state is : " + appHandle.getState());
> >         if (appHandle.getAppId() != null) {
> >             log.info("\t App id: " + appHandle.getAppId() + "\tState: "
> +
> > appHandle.getState());
> >         }
> >     }
> > }
>
> This is a ridiculously expensive busy loop, even more so if you
> comment out the log lines. Use listeners, or at least sleep a little
> bit every once in a while. You're probably starving other processes /
> threads of cpu.
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Mime
View raw message