spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zachary S Ennenga (Jira)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-28889) Allow UDTs to define custom casting behavior
Date Wed, 28 Aug 2019 01:34:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-28889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917341#comment-16917341
] 

Zachary S Ennenga commented on SPARK-28889:
-------------------------------------------

I don't know the intent behind some of the existing UDT casting mechanisms (particularly UDT
to string). If someone wants to weigh in on the impact that allowing users to override those
would have, that would be great. 

> Allow UDTs to define custom casting behavior
> --------------------------------------------
>
>                 Key: SPARK-28889
>                 URL: https://issues.apache.org/jira/browse/SPARK-28889
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.4.3
>            Reporter: Zachary S Ennenga
>            Priority: Minor
>
> Looking at `org.apache.spark.sql.catalyst.expressions.Cast`, UDTs do not support any
sort of casting except for identity casts, IE:
> {code:java}
> case (udt1: UserDefinedType[_], udt2: UserDefinedType[_]) if udt1.userClass == udt2.userClass
=>
>  true
> {code}
> I propose we add an additional piece of functionality here to allow UDTs to define their
own canCast and cast functions to allow users to define their own cast mechanisms.
> An example of how this might look:
> {code:java}
> case (fromType, toType: UserDefinedType[_]) =>
>  toType.canCast(fromType) // Returns boolean
> {code}
> {code:java}
> case (fromType, toType: UserDefinedType[_]) =>
>  toType.cast(fromType) // Returns Casting function
> {code}
> The UDT base class would contain a default implementation that replicates current behavior
(IE no casting).



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message