spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jianshi Huang <jianshi.hu...@gmail.com>
Subject Re: Hive UDAF percentile_approx says "This UDAF does not support the deprecated getEvaluator() method."
Date Tue, 13 Jan 2015 23:28:26 GMT
Ah, thx Ted and Yin!

I'll build a new version. :)

Jianshi

On Wed, Jan 14, 2015 at 7:24 AM, Yin Huai <yhuai@databricks.com> wrote:

> Yeah, it's a bug. It has been fixed by
> https://issues.apache.org/jira/browse/SPARK-3891 in master.
>
> On Tue, Jan 13, 2015 at 2:41 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
>> Looking at the source code for AbstractGenericUDAFResolver, the following
>> (non-deprecated) method should be called:
>>
>>   public GenericUDAFEvaluator getEvaluator(GenericUDAFParameterInfo info)
>>
>> It is called by hiveUdfs.scala (master branch):
>>
>>     val parameterInfo = new
>> SimpleGenericUDAFParameterInfo(inspectors.toArray, false, false)
>>     resolver.getEvaluator(parameterInfo)
>>
>> FYI
>>
>> On Tue, Jan 13, 2015 at 1:51 PM, Jianshi Huang <jianshi.huang@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> The following SQL query
>>>
>>> select percentile_approx(variables.var1, 0.95) p95
>>> from model
>>>
>>> will throw
>>>
>>> ERROR SparkSqlInterpreter: Error
>>> org.apache.hadoop.hive.ql.parse.SemanticException: This UDAF does not
>>> support the deprecated getEvaluator() method.
>>>         at
>>> org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:53)
>>>         at
>>> org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector$lzycompute(hiveUdfs.scala:196)
>>>         at
>>> org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector(hiveUdfs.scala:195)
>>>         at
>>> org.apache.spark.sql.hive.HiveGenericUdaf.dataType(hiveUdfs.scala:203)
>>>         at
>>> org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:105)
>>>         at
>>> org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143)
>>>         at
>>> org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143)
>>>
>>> I'm using latest branch-1.2
>>>
>>> I found in PR that percentile and percentile_approx are supported. A bug?
>>>
>>> Thanks,
>>> --
>>> Jianshi Huang
>>>
>>> LinkedIn: jianshi
>>> Twitter: @jshuang
>>> Github & Blog: http://huangjs.github.com/
>>>
>>
>>
>


-- 
Jianshi Huang

LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/

Mime
View raw message