spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Beavers <cbeav...@trifacta.com>
Subject UDF returning generic Seq
Date Tue, 26 Jul 2016 02:32:12 GMT
Hey there,

Interested in writing a UDF that returns an ArrayType column of unknown
subtype. My understanding is that this translated JVM-type-wise be a Seq of
generic templated type: Seq[Any]. I seem to be hitting the constraint at
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala:657
that
basically necessitates a fully qualified schema on the return type (i.e.
the templated Any is hitting the default exception throwing case at the end
of schemaFor).

Is there any more canonical way have a UDF produce an ArrayType column of
unknown type? Or is my only alternative here to reduce this to BinaryType
and use whatever encoding/data structures I want under the covers there and
in subsequent UDFs?

Thanks,
Chris

Mime
View raw message