Hi Aakash,

You can try the latest DS v2 with the 3.0 preview, and the API is in a quite stable shape now. With the latest API, a Writer is created from a Table, and the Table has the partitioning information.

Thanks,
Wenchen

On Wed, Dec 18, 2019 at 3:22 AM aakash aakash <email2aakash@gmail.com> wrote:
Thanks Andrew!

It seems there is a drastic change in 3.0, going through it.

-Aakash

On Tue, Dec 17, 2019 at 11:01 AM Andrew Melo <andrew.melo@gmail.com> wrote:
Hi Aakash

On Tue, Dec 17, 2019 at 12:42 PM aakash aakash <email2aakash@gmail.com> wrote:
Hi Spark dev folks,

First of all kudos on this new Data Source v2, API looks simple and it makes easy to develop a new data source and use it.

With my current work, I am trying to implement a new data source V2 writer with Spark 2.3 and I was wondering how I will get the info about partition by columns. I see that it has been passed to Data Source V1 from DataFrameWriter but not for V2.

Not directly related to your Q, but just so you're aware, the DSv2 API evolved from 2.3->2.4 and then again for 2.4->3.0.

Cheers
Andrew
 


Thanks,
Aakash