spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jungtaek Lim <>
Subject [DISCUSS] Time to evaluate "continuous mode" in SS?
Date Tue, 15 Sep 2020 04:34:04 GMT
Hi devs,

It was Spark 2.3 in Feb 2018 which introduced continuous mode in Structured
Streaming as "experimental".

Now we are here at 2.5 years after its release - I feel it would be a good
time to evaluate the mode, whether the mode has been widely used or not,
and the mode has been making progress, as the mode is "experimental".

At least from the surface I don't see any active effort for continuous mode
around the community - the last major effort was stateful operation which
was incomplete and I removed that. There were some couples of bug reports
as well as fixes more than a year ago and almost nothing has been handled.
(A trivial bugfix PR has been merged recently but that's all.) The new
features introduced to the Structured Streaming (at least observable
metrics, SS UI) don't apply to continuous mode, and no one made "support
continuous mode" as a hard requirement on passing review in these PRs.

I have no idea how many companies are using the mode in production (please
add the voice if someone has statistics about this) but I don't see any bug
reports recently, and see only a few questions in SO, which makes me think
about cost on maintenance.

I know there's a mood to avoid discontinue support as possible, but it
sounds weird to keep something as "unmaintained", especially it's still
"experimental" and main authors are no more active enough to promise
maintenance/improvement on the module. Thoughts?

Jungtaek Lim (HeartSaVioR)

View raw message