spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ε–œδΉ‹ιƒŽ" <251922...@qq.com>
Subject spark udf can not change a json string to a map
Date Sun, 15 May 2016 16:18:38 GMT
Hi, all. I want to implement a udf which is used to change a json string to a map<string,string>.
But some problem occurs. My spark version:1.5.1.




my udf code:
####################
	public Map<String,String> evaluate(final String s) {
		if (s == null)
			return null;
		return getString(s);
	}


	@SuppressWarnings("unchecked")
	public static Map<String,String> getString(String s) {
		try {
			String str =  URLDecoder.decode(s, "UTF-8");
			ObjectMapper mapper = new ObjectMapper();
			Map<String,String>  map = mapper.readValue(str, Map.class);
			
			return map;
		} catch (Exception e) {
			return new HashMap<String,String>();
		}
	}

#############
exception infos:


16/05/14 21:05:22 ERROR CliDriver: org.apache.spark.sql.AnalysisException: Map type in java
is unsupported because JVM type erasure makes spark fail to catch key and value types in Map<>;
line 1 pos 352
	at org.apache.spark.sql.hive.HiveInspectors$class.javaClassToDataType(HiveInspectors.scala:230)
	at org.apache.spark.sql.hive.HiveSimpleUDF.javaClassToDataType(hiveUDFs.scala:107)
	at org.apache.spark.sql.hive.HiveSimpleUDF.<init>(hiveUDFs.scala:136)

################




I have saw that there is a testsuite in spark says spark did not support this kind of udf.
But is there a method to implement this udf?
Mime
View raw message