spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jungtaek Lim <>
Subject Re: My curation of pending structured streaming PRs to review
Date Mon, 19 Aug 2019 03:08:37 GMT
As a reminder, the list contains two correctness bugs: stream-stream outer
join, and multiple stateful operations with watermark.

Regarding common theme, yes that's somewhat I'd rather avoid to say, but
honestly I feel there's shortage on active committers on 'structured

Many of them I know as relevant to SS area didn't show up themselves in
Spark community for around half a year (maybe even more), and unfortunately
even active committers seem to have struggled with shortage of time doing
their own works (that's natural) and haven't found time to focus reviewing
other PRs (provide valuable comments but not leading PRs as shepherd to be
merged). I hoped that's temporary issue for some important events like
Spark+AI summit, and turned out it's not.

Spark has no replacement of SS, DStream is now even cared less than SS.
Does Spark community not feeling important from streaming area? I might not
agree, as there're reports from end users and patches proposed so far from
contributors. I wouldn't the right one to say how can solve the issue, but
I hope we would handle the main issue nicely and less painful way.

On Tue, Aug 13, 2019 at 10:42 PM Sean Owen <> wrote:

> General tips:
> - dev@ is not usually the right place to discuss _specific_ changes
> except once in a while to call attention
> - Ping the authors of the code being changed directly
> - Tighten the change if possible
> - Tests, reproductions, docs, etc help prove the change
> - Bugs are more important than new marginal features
> If there has been some feedback that's just skeptical about the
> approach or value, that may be the answer, it won't be merged.
> If there is no feedback and it seems important (correctness bugs) it's
> OK to raise that here once in a while.
> One common theme here is 'structured streaming' -- who amongst the
> committers feels they are able to review these changes? I sense we
> have a shortage there.

Name : Jungtaek Lim
Blog :
Twitter :
LinkedIn :

View raw message