spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiao Li <>
Subject Re: Seeking committers' help to review on SS PR
Date Mon, 30 Nov 2020 20:08:30 GMT
Just want to say thank you to all the active SS contributors. I saw many
great features/improvements in Streaming have been merged and will be
available in the upcoming 3.1 release.

   - Cache fetched list of files beyond maxFilesPerTrigger as unread file
   - Streamline the logic on file stream source and sink metadata log
   - Add DataStreamReader.table API (SPARK-32885)
   - Add DataStreamWriter.saveAsTable API (SPARK-32896)
   - Left semi stream-stream join (SPARK-32862)
   - Introduce schema validation for streaming state store (SPARK-31894)
   - Support to use a different compression codec in state store
   - Kafka connector infinite wait because metadata never updated
   - Upgrade Kafka to 2.6.0 (SPARK-32568)
   - Pagination support for Structured Streaming UI pages (SPARK-31642,
   - State information in Structured Streaming UI (SPARK-33223)

Structured Streaming UI support in Spark History Server is another great
usability feature: Hopefully,
this can be part of 3.1 release.

Go Spark!


Ryan Blue <> 于2020年11月30日周一 上午11:35写道:

> Jungtaek,
> If there are contributors that you trust for reviews, then please let PMC
> members know so they can be considered. I agree that is the best solution.
> If there aren't contributors that the PMC wants to add as committers, then
> I suggest agreeing on a temporary exception to help make progress in this
> area and give contributors more opportunities to develop. Something like
> this: for the next 6 months, contributions from committers to SS can be
> committed without a committer +1 if they are reviewed by at least one
> contributor (and have no dissent from committers, of course). Then after
> the period expires, we would ideally have new people ready to be added as
> committers.
> That would need to be voted on, but I think it is a reasonable step to
> help resuscitate Spark streaming.
> On Fri, Nov 27, 2020 at 7:15 PM Sean Owen <> wrote:
>> I don't know the code well, but those look minor and straightforward.
>> They have reviews from the two most knowledgeable people in this area. I
>> don't think you need to block for 6 months after proactively seeking all
>> likely reviewers - I'm saying that's the resolution to this type of
>> situation (too).
>> On Fri, Nov 27, 2020 at 8:55 PM Jungtaek Lim <
>>> wrote:
>>> Btw, there are two more PRs which got LGTM by a SS contributor but fail
>>> to get attention from committers. They're 6+ months old. Could you help
>>> reviewing this as well, or do you all think 6 months of time range + LGTM
>>> from an SS contributor is enough to go ahead?
>>> These are under 100 lines of changes per each, and not invasive.
> --
> Ryan Blue
> Software Engineer
> Netflix

View raw message