spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <jornfra...@gmail.com>
Subject Re: Spark 2 + Java + UDF + unknown return type...
Date Thu, 02 Feb 2017 21:26:18 GMT
 Not sure what your udf is exactly doing, but why not on udf / type ? You avoid issues converting
it, it is more obvious for the user of your udf etc
You could of course return a complex type with one long, one string and one double and you
fill them in the udf as needed, but this would be probably not a clean solution...

> On 2 Feb 2017, at 22:05, Jean Georges Perrin <jgp@jgp.net> wrote:
> 
> Hi fellow Sparkans,
> 
> I am building a UDF (in Java) that can return various data types, basically the signature
of the function itself is:
> 
> 	public Object call(String a, Object b, String c, Object d, String e) throws Exception
> 
> When I register my function, I need to provide a type, e.g.:
> 
> 	spark.udf().register("f2", new Udf5(), DataTypes.LongType);
> 
> In my test it is a long now, but can become a string or a float. Of course, I do not
know the expected return type before I call the function, which I call like:
> 
> 	df = df.selectExpr("*", "f2('x1', x, 'c2', y, 'op') as op");
> 
> Is there a way to have an Object being returned from a UDF and to store an Object in
a Dataset/dataframe? I don't need to know the datatype at that point and can leave it hanging
for now? Or should I play it safe and always return a DataTypes.StringType (and then try to
transform it if needed)?
> 
> I hope I am clear enough :).
> 
> Thanks for any tip/idea/comment...
> 
> jg

Mime
View raw message