spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chao Sun <sunc...@apache.org>
Subject Re: [VOTE] SPIP: Add FunctionCatalog
Date Tue, 09 Mar 2021 01:20:25 GMT
+1 (non-binding)

On Mon, Mar 8, 2021 at 5:13 PM John Zhuge <jzhuge@apache.org> wrote:

> +1 (non-binding)
>
> On Mon, Mar 8, 2021 at 4:32 PM Holden Karau <holden@pigscanfly.ca> wrote:
>
>> +1 (binding)
>>
>> On Mon, Mar 8, 2021 at 3:56 PM Ryan Blue <blue@apache.org> wrote:
>>
>>> Hi everyone, I’d like to start a vote for the FunctionCatalog design
>>> proposal (SPIP).
>>>
>>> The proposal is to add a FunctionCatalog interface that can be used to
>>> load and list functions for Spark to call. There are interfaces for scalar
>>> and aggregate functions.
>>>
>>> In the discussion we’ve come to consensus and I’ve updated the design
>>> doc to match how functions will be called:
>>>
>>> In addition to produceResult(InternalRow), which is optional, functions
>>> can define produceResult methods with arguments that are Spark’s
>>> internal data types, like UTF8String. Spark will prefer these methods
>>> when calling the UDF using codgen.
>>>
>>> I’ve also updated the AggregateFunction interface and merged it with
>>> the partial aggregate interface because Spark doesn’t support non-partial
>>> aggregates.
>>>
>>> The full SPIP doc is here:
>>> https://docs.google.com/document/d/1PLBieHIlxZjmoUB0ERF-VozCRJ0xw2j3qKvUNWpWA2U/edit#heading=h.82w8qxfl2uwl
>>>
>>> Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll
>>> do a final update of the PR and we can merge the API.
>>>
>>> [ ] +1: Accept the proposal as an official SPIP
>>> [ ] +0
>>> [ ] -1: I don’t think this is a good idea because …
>>> --
>>> Ryan Blue
>>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>
>
> --
> John Zhuge
>

Mime
View raw message