spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheng, Hao" <hao.ch...@intel.com>
Subject RE: SparkSQL : using Hive UDF returning Map throws "rror: scala.MatchError: interface java.util.Map (of class java.lang.Class) (state=,code=0)"
Date Fri, 05 Jun 2015 07:17:38 GMT
Confirmed, with latest master, we don't support complex data type for Simple Hive UDF, do you
mind file an issue in jira?

-----Original Message-----
From: Cheng, Hao [mailto:hao.cheng@intel.com] 
Sent: Friday, June 5, 2015 12:35 PM
To: ogoh; user@spark.apache.org
Subject: RE: SparkSQL : using Hive UDF returning Map throws "rror: scala.MatchError: interface
java.util.Map (of class java.lang.Class) (state=,code=0)"

Which version of Hive jar are you using? Hive 0.13.1 or Hive 0.12.0?

-----Original Message-----
From: ogoh [mailto:okehee@gmail.com] 
Sent: Friday, June 5, 2015 10:10 AM
To: user@spark.apache.org
Subject: SparkSQL : using Hive UDF returning Map throws "rror: scala.MatchError: interface
java.util.Map (of class java.lang.Class) (state=,code=0)"


Hello,
I tested some custom udf on SparkSql's ThriftServer & Beeline (Spark 1.3.1).
Some udfs work fine (access array parameter and returning int or string type). 
But my udf returning map type throws an error:
"Error: scala.MatchError: interface java.util.Map (of class java.lang.Class) (state=,code=0)"

I converted the code into Hive's GenericUDF since I worried that using complex type parameter
(array of map) and returning complex type (map) can be supported in Hive's GenericUDF instead
of simple UDF.
But SparkSQL doesn't seem supporting GenericUDF.(error message : Error:
java.lang.IllegalAccessException: Class
org.apache.spark.sql.hive.HiveFunctionWrapper can not access ..).

Below is my example udf code returning MAP type.
I appreciate any advice.
Thanks

------

public final class ArrayToMap extends UDF {

    public Map<String,String> evaluate(ArrayList<String> arrayOfString) {
        // add code to handle all index problem

        Map<String, String> map = new HashMap<String, String>();
       
        int count = 0;
        for (String element : arrayOfString) {
            map.put(count + "", element);
            count++;
            
        }
        return map;
    }
}






--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkSQL-using-Hive-UDF-returning-Map-throws-rror-scala-MatchError-interface-java-util-Map-of-class--tp23164.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail:
user-help@spark.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message