spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gourav Sengupta <>
Subject Re: Structured Streaming: How to add a listener for when a batch is complete
Date Wed, 04 Sep 2019 15:02:55 GMT
Hi Natalie,
in case we are using Python then listener is not available, we can use the

For that to happen we have to start the SPARK session with following
settings: spark.sql("SET spark.sql.streaming.metricsEnabled=true")
For  the UI from which we can use the JSON results from the below:

In case you are working on AWS EMR, the UI URL is a bit different, let me
know in that case and I will email to you.

Hi TD,
What will be the best way to stop a long running SPARK streaming job
gracefully? I will be really grateful if you could kindly help us out.

Gourav Sengupta

On Wed, Sep 4, 2019 at 2:28 AM Tathagata Das <> wrote:

> On Tue, Sep 3, 2019, 3:26 PM Natalie Ruiz
> <> wrote:
>> Hello all,
>> I’m a beginner, new to Spark and wanted to know if there was an
>> equivalent to Spark Streaming’s StreamingListenerBatchCompleted in
>> Structured Streaming? I want to add a listener for when a batch is complete
>> but the documentation and examples I find are for Spark Streaming and not
>> Structured Streaming and registering it to a StreamingContext, which from
>> my understanding is just for Spark Streaming
>> Thanks!

View raw message