spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean Georges Perrin <>
Subject Spark 2 + Java + UDF + unknown return type...
Date Thu, 02 Feb 2017 21:05:39 GMT
Hi fellow Sparkans,

I am building a UDF (in Java) that can return various data types, basically the signature
of the function itself is:

	public Object call(String a, Object b, String c, Object d, String e) throws Exception

When I register my function, I need to provide a type, e.g.:

	spark.udf().register("f2", new Udf5(), DataTypes.LongType);

In my test it is a long now, but can become a string or a float. Of course, I do not know
the expected return type before I call the function, which I call like:

	df = df.selectExpr("*", "f2('x1', x, 'c2', y, 'op') as op");

Is there a way to have an Object being returned from a UDF and to store an Object in a Dataset/dataframe?
I don't need to know the datatype at that point and can leave it hanging for now? Or should
I play it safe and always return a DataTypes.StringType (and then try to transform it if needed)?

I hope I am clear enough :).

Thanks for any tip/idea/comment...

View raw message