spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Armbrust (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-3904) HQL doesn't support the ConstantObjectInspector to pass into UDFs
Date Wed, 29 Oct 2014 02:12:33 GMT

     [ https://issues.apache.org/jira/browse/SPARK-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Michael Armbrust resolved SPARK-3904.
-------------------------------------
       Resolution: Fixed
    Fix Version/s: 1.2.0

Issue resolved by pull request 2762
[https://github.com/apache/spark/pull/2762]

> HQL doesn't support the ConstantObjectInspector to pass into UDFs
> -----------------------------------------------------------------
>
>                 Key: SPARK-3904
>                 URL: https://issues.apache.org/jira/browse/SPARK-3904
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>            Reporter: Cheng Hao
>            Assignee: Cheng Hao
>             Fix For: 1.2.0
>
>
> In HQL, we convert all of the data type into normal ObjectInspectors for UDFs, most of
cases it work, however, some of the UDF actually requires the input ObjectInspector to be
the ConstantObjectInspector, which will cause exception.
> e.g.
> {panel}
> select named_struct("x", "str") from src limit 1
> {panel}
> It will throws exception like
> {panel}
> 14/10/10 16:25:17 INFO parse.ParseDriver: Parsing command: select named_struct("x", "str")
from src
> 14/10/10 16:25:17 INFO parse.ParseDriver: Parse Completed
> 14/10/10 16:25:17 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=src
> 14/10/10 16:25:17 INFO HiveMetaStore.audit: ugi=hcheng	ip=unknown-ip-addr	cmd=get_table
: db=default tbl=tmp2	
> 14/10/10 16:25:17 ERROR thriftserver.SparkSQLDriver: Failed in [select named_struct("x",
"str") from src]
> org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: Even arguments to NAMED_STRUCT
must be a constant STRING.org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaStringObjectInspector@2f2dbcfc
> 	at org.apache.hadoop.hive.ql.udf.generic.GenericUDFNamedStruct.initialize(GenericUDFNamedStruct.java:55)
> 	at org.apache.spark.sql.hive.HiveGenericUdf.returnInspector$lzycompute(hiveUdfs.scala:129)
> 	at org.apache.spark.sql.hive.HiveGenericUdf.returnInspector(hiveUdfs.scala:129)
> 	at org.apache.spark.sql.hive.HiveGenericUdf.eval(hiveUdfs.scala:158)
> 	at org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$6$$anonfun$applyOrElse$2.applyOrElse(Optimizer.scala:267)
> 	at org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$6$$anonfun$applyOrElse$2.applyOrElse(Optimizer.scala:260)
> {panel}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message