spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Okehee Goh <oke...@gmail.com>
Subject Re: SparkSQL : using Hive UDF returning Map throws "rror: scala.MatchError: interface java.util.Map (of class java.lang.Class) (state=,code=0)"
Date Fri, 05 Jun 2015 07:17:51 GMT
It is Spark 1.3.1.e (it is AWS release .. I think it is close to Spark
1.3.1 with some bug fixes).

My report about GenericUDF not working in SparkSQL is wrong. I tested
with open-source GenericUDF and it worked fine. Just my GenericUDF
which returns Map type didn't work. Sorry about false reporting.



On Thu, Jun 4, 2015 at 9:35 PM, Cheng, Hao <hao.cheng@intel.com> wrote:
> Which version of Hive jar are you using? Hive 0.13.1 or Hive 0.12.0?
>
> -----Original Message-----
> From: ogoh [mailto:okehee@gmail.com]
> Sent: Friday, June 5, 2015 10:10 AM
> To: user@spark.apache.org
> Subject: SparkSQL : using Hive UDF returning Map throws "rror: scala.MatchError: interface
java.util.Map (of class java.lang.Class) (state=,code=0)"
>
>
> Hello,
> I tested some custom udf on SparkSql's ThriftServer & Beeline (Spark 1.3.1).
> Some udfs work fine (access array parameter and returning int or string type).
> But my udf returning map type throws an error:
> "Error: scala.MatchError: interface java.util.Map (of class java.lang.Class) (state=,code=0)"
>
> I converted the code into Hive's GenericUDF since I worried that using complex type parameter
(array of map) and returning complex type (map) can be supported in Hive's GenericUDF instead
of simple UDF.
> But SparkSQL doesn't seem supporting GenericUDF.(error message : Error:
> java.lang.IllegalAccessException: Class
> org.apache.spark.sql.hive.HiveFunctionWrapper can not access ..).
>
> Below is my example udf code returning MAP type.
> I appreciate any advice.
> Thanks
>
> ------
>
> public final class ArrayToMap extends UDF {
>
>     public Map<String,String> evaluate(ArrayList<String> arrayOfString) {
>         // add code to handle all index problem
>
>         Map<String, String> map = new HashMap<String, String>();
>
>         int count = 0;
>         for (String element : arrayOfString) {
>             map.put(count + "", element);
>             count++;
>
>         }
>         return map;
>     }
> }
>
>
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkSQL-using-Hive-UDF-returning-Map-throws-rror-scala-MatchError-interface-java-util-Map-of-class--tp23164.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail:
user-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message