spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aakash aakash <email2aak...@gmail.com>
Subject Re: how to get partition column info in Data Source V2 writer
Date Thu, 19 Dec 2019 04:31:28 GMT
Thanks Wenchen!

On Wed, Dec 18, 2019 at 7:25 PM Wenchen Fan <cloud0fan@gmail.com> wrote:

> Hi Aakash,
>
> You can try the latest DS v2 with the 3.0 preview, and the API is in a
> quite stable shape now. With the latest API, a Writer is created from a
> Table, and the Table has the partitioning information.
>
> Thanks,
> Wenchen
>
> On Wed, Dec 18, 2019 at 3:22 AM aakash aakash <email2aakash@gmail.com>
> wrote:
>
>> Thanks Andrew!
>>
>> It seems there is a drastic change in 3.0, going through it.
>>
>> -Aakash
>>
>> On Tue, Dec 17, 2019 at 11:01 AM Andrew Melo <andrew.melo@gmail.com>
>> wrote:
>>
>>> Hi Aakash
>>>
>>> On Tue, Dec 17, 2019 at 12:42 PM aakash aakash <email2aakash@gmail.com>
>>> wrote:
>>>
>>>> Hi Spark dev folks,
>>>>
>>>> First of all kudos on this new Data Source v2, API looks simple and it
>>>> makes easy to develop a new data source and use it.
>>>>
>>>> With my current work, I am trying to implement a new data source V2
>>>> writer with Spark 2.3 and I was wondering how I will get the info about
>>>> partition by columns. I see that it has been passed to Data Source V1 from
>>>> DataFrameWriter but not for V2.
>>>>
>>>
>>> Not directly related to your Q, but just so you're aware, the DSv2 API
>>> evolved from 2.3->2.4 and then again for 2.4->3.0.
>>>
>>> Cheers
>>> Andrew
>>>
>>>
>>>>
>>>>
>>>> Thanks,
>>>> Aakash
>>>>
>>>

Mime
View raw message