spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Georg Heiler <georg.kf.hei...@gmail.com>
Subject Re: Custom Window Function
Date Wed, 23 Jan 2019 16:16:21 GMT
Hi Herman,

Thanks a lot. So far I only found most of the documentation about UDAF.
Could you point me anywhere (besides just reading spark's source code)
which explains how to work with custom AggregateFunctions?

Best,
Georg

Am Mi., 23. Jan. 2019 um 16:02 Uhr schrieb Herman van Hovell <
herman@databricks.com>:

> Hi Georg,
>
> In most cases you want to implement an aggregate function. You can either
> define a UDAF, or an AggregateFunction. The latter is an internal
> expression, so it is faster but we might break it across versions of spark.
>
> Hope this helps.
>
> Cheers,
> Herman
>
> On Wed, Jan 23, 2019 at 3:55 PM Georg Heiler <georg.kf.heiler@gmail.com>
> wrote:
>
>> Hi,
>>
>> I want to write custom window functions in spark which are also
>> optimisable for catalyst.
>> Can you provide some hints where to start?
>>
>> Also posting to DEVLIST as I believe this is a rather exotic topic.
>>
>> Best,
>> Georg
>>
>

Mime
View raw message