Thanks Andrew!

It seems there is a drastic change in 3.0, going through it.

-Aakash

On Tue, Dec 17, 2019 at 11:01 AM Andrew Melo <andrew.melo@gmail.com> wrote:
Hi Aakash

On Tue, Dec 17, 2019 at 12:42 PM aakash aakash <email2aakash@gmail.com> wrote:
Hi Spark dev folks,

First of all kudos on this new Data Source v2, API looks simple and it makes easy to develop a new data source and use it.

With my current work, I am trying to implement a new data source V2 writer with Spark 2.3 and I was wondering how I will get the info about partition by columns. I see that it has been passed to Data Source V1 from DataFrameWriter but not for V2.

Not directly related to your Q, but just so you're aware, the DSv2 API evolved from 2.3->2.4 and then again for 2.4->3.0.

Cheers
Andrew
 


Thanks,
Aakash