spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wenchen Fan <cloud0...@gmail.com>
Subject Re: how to get partition column info in Data Source V2 writer
Date Thu, 19 Dec 2019 03:25:41 GMT
Hi Aakash,

You can try the latest DS v2 with the 3.0 preview, and the API is in a
quite stable shape now. With the latest API, a Writer is created from a
Table, and the Table has the partitioning information.

Thanks,
Wenchen

On Wed, Dec 18, 2019 at 3:22 AM aakash aakash <email2aakash@gmail.com>
wrote:

> Thanks Andrew!
>
> It seems there is a drastic change in 3.0, going through it.
>
> -Aakash
>
> On Tue, Dec 17, 2019 at 11:01 AM Andrew Melo <andrew.melo@gmail.com>
> wrote:
>
>> Hi Aakash
>>
>> On Tue, Dec 17, 2019 at 12:42 PM aakash aakash <email2aakash@gmail.com>
>> wrote:
>>
>>> Hi Spark dev folks,
>>>
>>> First of all kudos on this new Data Source v2, API looks simple and it
>>> makes easy to develop a new data source and use it.
>>>
>>> With my current work, I am trying to implement a new data source V2
>>> writer with Spark 2.3 and I was wondering how I will get the info about
>>> partition by columns. I see that it has been passed to Data Source V1 from
>>> DataFrameWriter but not for V2.
>>>
>>
>> Not directly related to your Q, but just so you're aware, the DSv2 API
>> evolved from 2.3->2.4 and then again for 2.4->3.0.
>>
>> Cheers
>> Andrew
>>
>>
>>>
>>>
>>> Thanks,
>>> Aakash
>>>
>>

Mime
View raw message