spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aakash aakash <email2aak...@gmail.com>
Subject Re: how to get partition column info in Data Source V2 writer
Date Tue, 17 Dec 2019 19:22:25 GMT
Thanks Andrew!

It seems there is a drastic change in 3.0, going through it.

-Aakash

On Tue, Dec 17, 2019 at 11:01 AM Andrew Melo <andrew.melo@gmail.com> wrote:

> Hi Aakash
>
> On Tue, Dec 17, 2019 at 12:42 PM aakash aakash <email2aakash@gmail.com>
> wrote:
>
>> Hi Spark dev folks,
>>
>> First of all kudos on this new Data Source v2, API looks simple and it
>> makes easy to develop a new data source and use it.
>>
>> With my current work, I am trying to implement a new data source V2
>> writer with Spark 2.3 and I was wondering how I will get the info about
>> partition by columns. I see that it has been passed to Data Source V1 from
>> DataFrameWriter but not for V2.
>>
>
> Not directly related to your Q, but just so you're aware, the DSv2 API
> evolved from 2.3->2.4 and then again for 2.4->3.0.
>
> Cheers
> Andrew
>
>
>>
>>
>> Thanks,
>> Aakash
>>
>

Mime
View raw message