spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Felix Cheung <felixcheun...@hotmail.com>
Subject Re: [VOTE] SPIP: Spark API for Table Metadata
Date Sun, 03 Mar 2019 06:13:02 GMT
+1

We reviewed and discussed this for quite some time.


________________________________
From: Xiao Li <lixiao@databricks.com>
Sent: Friday, March 1, 2019 7:32 PM
To: Wenchen Fan
Cc: Reynold Xin; Ryan Blue; Spark Dev List
Subject: Re: [VOTE] SPIP: Spark API for Table Metadata

+1

Thanks for your efforts! The high-level direction is fine to me. Let us discuss the details
of APIs in the design stage and PR reviews.

Xiao

On Fri, Mar 1, 2019 at 6:21 PM Wenchen Fan <cloud0fan@gmail.com<mailto:cloud0fan@gmail.com>>
wrote:
+1, thanks for making it clear that this SPIP focuses on high-level direction!

On Sat, Mar 2, 2019 at 9:35 AM Reynold Xin <rxin@databricks.com<mailto:rxin@databricks.com>>
wrote:

Thanks Ryan. +1.




On Fri, Mar 01, 2019 at 5:33 PM, Ryan Blue <rblue@netflix.com<mailto:rblue@netflix.com>>
wrote:
Actually, I went ahead and removed the confusing section. There is no public API in the doc
now, so that it is clear that it isn't a relevant part of this vote.

On Fri, Mar 1, 2019 at 4:58 PM Ryan Blue <rblue@netflix.com<mailto:rblue@netflix.com>>
wrote:
I moved the public API to the "Implementation Sketch" section. That API is not an important
part of this, as that section notes.

I completely agree that SPIPs should be high-level and that the specifics, like method names,
are not hard requirements. The proposal was more of a sketch, but I was asked by Xiao in the
DSv2 sync to make sure the list of methods was complete. I think as long as we have agreement
that the intent is not to make the exact names binding, we should be okay.

I can remove the user-facing API sketch, but I'd prefer to leave it in the sketch section
so we have it documented somewhere.

On Fri, Mar 1, 2019 at 4:51 PM Reynold Xin <rxin@databricks.com<mailto:rxin@databricks.com>>
wrote:
Ryan - can you take the public user facing API part out of that SPIP?

In general it'd be better to have the SPIPs be higher level, and put the detailed APIs in
a separate doc. Alternatively, put them in the SPIP but explicitly vote on the high level
stuff and not the detailed APIs.

I don't want to get to a situation in which two months later the identical APIs were committed
with the justification that they were voted on a while ago. In this case, it's even more serious
because while I think we all have consensus on the higher level internal API, not much discussion
has happened with the user-facing API and we should just leave that out explicitly.






On Fri, Mar 01, 2019 at 1:00 PM, Anthony Young-Garner <anthony.young-garner@cloudera.com.invalid<mailto:anthony.young-garner@cloudera.com.invalid>>
wrote:
+1 (non-binding)

On Thu, Feb 28, 2019 at 5:54 PM John Zhuge <jzhuge@apache.org<mailto:jzhuge@apache.org>>
wrote:
+1 (non-binding)

On Thu, Feb 28, 2019 at 9:11 AM Matt Cheah <mcheah@palantir.com<mailto:mcheah@palantir.com>>
wrote:

+1 (non-binding)



From:Jamison Bennett <jamison.bennett@cloudera.com.INVALID<mailto:jamison.bennett@cloudera.com.INVALID>>
Date:Thursday, February 28, 2019 at 8:28 AM
To:Ryan Blue <rblue@netflix.com<mailto:rblue@netflix.com>>, Spark Dev List <dev@spark.apache.org<mailto:dev@spark.apache.org>>
Subject:Re: [VOTE] SPIP: Spark API for Table Metadata



+1 (non-binding)


Jamison Bennett

Cloudera Software Engineer

jamison.bennett@cloudera.com<mailto:jamison.bennett@cloudera.com>

515 Congress Ave, Suite 1212   |   Austin, TX   |   78701





On Thu, Feb 28, 2019 at 10:20 AM Ryan Blue <rblue@netflix.com.invalid<mailto:rblue@netflix.com.invalid>>
wrote:

+1 (non-binding)



On Wed, Feb 27, 2019 at 8:34 PM Russell Spitzer <russell.spitzer@gmail.com<mailto:russell.spitzer@gmail.com>>
wrote:

+1 (non-binding)

On Wed, Feb 27, 2019, 6:28 PM Ryan Blue <rblue@netflix.com.invalid<mailto:rblue@netflix.com.invalid>>
wrote:

Hi everyone,



In the last DSv2 sync, the consensus was that the table metadata SPIP was ready to bring up
for a vote. Now that the multi-catalog identifier SPIP vote has passed, I'd like to start
one for the table metadata API, TableCatalog.



The proposal is for adding a TableCatalog interface that will be used by v2 plans. That interface
has methods to load, create, drop, alter, refresh, rename, and check existence for tables.
It also specifies the set of metadata used to configure tables: schema, partitioning, and
key-value properties. For more information, please read the SPIP proposal doc [docs.google.com]<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo_edit-23heading-3Dh.m45webtwxf2d&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=JmgvL6ffL9tyoLWWZtWujDe9FNiSguMApA53YK9NTP8&s=eSx5nMZvdB5hS9VepuvvFZFXjTCrdde-AdzkHC5jRYk&e=>.



Please vote in the next 3 days.



[ ] +1: Accept the proposal as an official SPIP

[ ] +0

[ ] -1: I don't think this is a good idea because ...





Thanks!



--

Ryan Blue

Software Engineer

Netflix




--

Ryan Blue

Software Engineer

Netflix


--
John Zhuge



--
Ryan Blue
Software Engineer
Netflix


--
Ryan Blue
Software Engineer
Netflix



--
[https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]

Mime
View raw message