spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wangzhenhua (G)" <>
Subject 答复: [VOTE] [SPIP] SPARK-15689: Data Source API V2 read path
Date Fri, 08 Sep 2017 02:20:07 GMT
+1 (non-binding)  Great to see data source API is going to be improved!

best regards,

发件人: Dongjoon Hyun []
发送时间: 2017年9月8日 4:07
收件人: 蒋星博
抄送: Michael Armbrust; Reynold Xin; Andrew Ash; Herman van Hövell tot Westerflier; Ryan
Blue; Spark dev list; Suresh Thalamati; Wenchen Fan
主题: Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2 read path

+1 (non-binding).

On Thu, Sep 7, 2017 at 12:46 PM, 蒋星博 <<>>

Reynold Xin <<>>于2017年9月7日
+1 as well

On Thu, Sep 7, 2017 at 9:12 PM, Michael Armbrust <<>>

On Thu, Sep 7, 2017 at 9:32 AM, Ryan Blue <<>>

+1 (non-binding)

Thanks for making the updates reflected in the current PR. It would be great to see the doc
updated before it is finally published though.

Right now it feels like this SPIP is focused more on getting the basics right for what many
datasources are already doing in API V1 combined with other private APIs, vs pushing forward
state of the art for performance.

I think that’s the right approach for this SPIP. We can add the support you’re talking
about later with a more specific plan that doesn’t block fixing the problems that this addresses.

On Thu, Sep 7, 2017 at 2:00 AM, Herman van Hövell tot Westerflier <<>>
+1 (binding)

I personally believe that there is quite a big difference between having a generic data source
interface with a low surface area and pushing down a significant part of query processing
into a datasource. The later has much wider wider surface area and will require us to stabilize
most of the internal catalyst API's which will be a significant burden on the community to
maintain and has the potential to slow development velocity significantly. If you want to
write such integrations then you should be prepared to work with catalyst internals and own
up to the fact that things might change across minor versions (and in some cases even maintenance
releases). If you are willing to go down that road, then your best bet is to use the already
existing spark session extensions which will allow you to write such integrations and can
be used as an `escape hatch`.

On Thu, Sep 7, 2017 at 10:23 AM, Andrew Ash <<>>
+0 (non-binding)

I think there are benefits to unifying all the Spark-internal datasources into a common public
API for sure.  It will serve as a forcing function to ensure that those internal datasources
aren't advantaged vs datasources developed externally as plugins to Spark, and that all Spark
features are available to all datasources.

But I also think this read-path proposal avoids the more difficult questions around how to
continue pushing datasource performance forwards.  James Baker (my colleague) had a number
of questions about advanced pushdowns (combined sorting and filtering), and Reynold also noted
that pushdown of aggregates and joins are desirable on longer timeframes as well.  The Spark
community saw similar requests, for aggregate pushdown in SPARK-12686, join pushdown in SPARK-20259,
and arbitrary plan pushdown in SPARK-12449.  Clearly a number of people are interested in
this kind of performance work for datasources.

To leave enough space for datasource developers to continue experimenting with advanced interactions
between Spark and their datasources, I'd propose we leave some sort of escape valve that enables
these datasources to keep pushing the boundaries without forking Spark.  Possibly that looks
like an additional unsupported/unstable interface that pushes down an entire (unstable API)
logical plan, which is expected to break API on every release.   (Spark attempts this full-plan
pushdown, and if that fails Spark ignores it and continues on with the rest of the V2 API
for compatibility).  Or maybe it looks like something else that we don't know of yet.  Possibly
this falls outside of the desired goals for the V2 API and instead should be a separate SPIP.

If we had a plan for this kind of escape valve for advanced datasource developers I'd be an
unequivocal +1.  Right now it feels like this SPIP is focused more on getting the basics right
for what many datasources are already doing in API V1 combined with other private APIs, vs
pushing forward state of the art for performance.


On Wed, Sep 6, 2017 at 10:56 PM, Suresh Thalamati <<>>
+1 (non-binding)

On Sep 6, 2017, at 7:29 PM, Wenchen Fan <<>>

Hi all,

In the previous discussion, we decided to split the read and write path of data source v2
into 2 SPIPs, and I'm sending this email to call a vote for Data Source V2 read path only.

The full document of the Data Source API V2 is:

The ready-for-review PR that implements the basic infrastructure for the read path is:

The vote will be up for the next 72 hours. Please reply with your vote:

+1: Yeah, let's go forward and implement the SPIP.
+0: Don't really care.
-1: I don't think this is a good idea because of the following technical reasons.



Herman van Hövell

Software Engineer

Databricks Inc.<>

+31 6 420 590 27<>


[Announcing Databricks Serverless. The first serverless data science and big data platform.
Watch the demo from Spark Summit 2017.]<>

Ryan Blue
Software Engineer

View raw message