spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yin Huai <yh...@databricks.com>
Subject Re: Hive UDAF percentile_approx says "This UDAF does not support the deprecated getEvaluator() method."
Date Tue, 13 Jan 2015 23:24:50 GMT
Yeah, it's a bug. It has been fixed by
https://issues.apache.org/jira/browse/SPARK-3891 in master.

On Tue, Jan 13, 2015 at 2:41 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> Looking at the source code for AbstractGenericUDAFResolver, the following
> (non-deprecated) method should be called:
>
>   public GenericUDAFEvaluator getEvaluator(GenericUDAFParameterInfo info)
>
> It is called by hiveUdfs.scala (master branch):
>
>     val parameterInfo = new
> SimpleGenericUDAFParameterInfo(inspectors.toArray, false, false)
>     resolver.getEvaluator(parameterInfo)
>
> FYI
>
> On Tue, Jan 13, 2015 at 1:51 PM, Jianshi Huang <jianshi.huang@gmail.com>
> wrote:
>
>> Hi,
>>
>> The following SQL query
>>
>> select percentile_approx(variables.var1, 0.95) p95
>> from model
>>
>> will throw
>>
>> ERROR SparkSqlInterpreter: Error
>> org.apache.hadoop.hive.ql.parse.SemanticException: This UDAF does not
>> support the deprecated getEvaluator() method.
>>         at
>> org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:53)
>>         at
>> org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector$lzycompute(hiveUdfs.scala:196)
>>         at
>> org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector(hiveUdfs.scala:195)
>>         at
>> org.apache.spark.sql.hive.HiveGenericUdaf.dataType(hiveUdfs.scala:203)
>>         at
>> org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:105)
>>         at
>> org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143)
>>         at
>> org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143)
>>
>> I'm using latest branch-1.2
>>
>> I found in PR that percentile and percentile_approx are supported. A bug?
>>
>> Thanks,
>> --
>> Jianshi Huang
>>
>> LinkedIn: jianshi
>> Twitter: @jshuang
>> Github & Blog: http://huangjs.github.com/
>>
>
>

Mime
View raw message