spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yuval Itzchakov <>
Subject Terminating Structured Streaming Applications on Source Failure
Date Tue, 29 Aug 2017 09:09:18 GMT
I wasn't sure if this would be a proper bug or not.

Today, the behavior of Structured Streaming is such that if a source fails
with an exception, the `StreamExecution` class halts reading further from
the source, but the application is remained alive. For applications where
the sole purpose is to transform data from a non static source (such as
Kafka), this is rather useless and might be surprising.

For example, if you have a simple monitor which checks whether the
application is alive or not, you'll still get reports that the application
is alive and running, but actually it isn't consuming anything from the
source and is logically dead.

Should this be the behavior? I think that perhaps there should be a
configuration that asks whether to completely shutdown the application on
source failure.

What do you guys think?

Best Regards,
Yuval Itzchakov.

View raw message