spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <tathagata.das1...@gmail.com>
Subject Re: spark streaming: what is awaitTermination()?
Date Thu, 27 Mar 2014 20:06:29 GMT
The execution of Spark Streaming (started with
StreamingContext.start()) can stop in two ways.
1. steamingContext.stop() is called (could be from a different thread)
2. some exception occurs in the processing of data.

awaitTermination is the right way for the main thread that started the
context to stay blocked, so that processing continues in the
background threads. The reason why removing awaitTermination is making
no difference because there is a bug in 0.9.0 that causes the main
function to not terminate even though the main thread has terminated
(one of the background thread is non-daemon). Also, without
awaitTermination, it is very hard to catch and print exceptions that
occur during the background data processing.

TD


On Thu, Mar 27, 2014 at 7:02 AM, Diana Carroll <dcarroll@cloudera.com> wrote:
> The API docs for ssc.awaitTermination say simply "Wait for the execution to
> stop. Any exceptions that occurs during the execution will be thrown in this
> thread."
>
> Can someone help me understand what this means?  What causes execution to
> stop?  Why do we need to wait for that to happen?
>
> I tried removing it from my simple NetworkWordCount example (running
> locally, not on a cluster) and nothing changed.  In both cases, I end my
> program by hitting Ctrl-C.
>
> Thanks for any insight you can give me.
>
> Diana

Mime
View raw message