spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-12264) Add a typeTag or scalaTypeTag method to DataType
Date Thu, 23 Feb 2017 15:10:44 GMT

    [ https://issues.apache.org/jira/browse/SPARK-12264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813946#comment-15813946
] 

Hyukjin Kwon edited comment on SPARK-12264 at 2/23/17 3:09 PM:
---------------------------------------------------------------

(I just simply changed the title to 

{quote}
add a typeTag or scalaTypeTag method to DataType.
{quote}
)


was (Author: hyukjin.kwon):
(I just simple change the title to 

{quote}
add a typeTag or scalaTypeTag method to DataType.
{quote}
)

> Add a typeTag or scalaTypeTag method to DataType
> ------------------------------------------------
>
>                 Key: SPARK-12264
>                 URL: https://issues.apache.org/jira/browse/SPARK-12264
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>            Reporter: Andras Nemeth
>            Priority: Minor
>
> We are writing code that's dealing with generic DataFrames as inputs and further processes
their contents with normal RDD operations (not SQL). We need some mechanism that tells us
exactly what Scala types we will find inside a Row of a given DataFrame.
> The schema of the DataFrame contains this information in an abstract sense. But we need
to map it to TypeTags, as that's what the rest of the system uses to identify what RDD contains
what type of data - quite the natural choice in Scala.
> As far as I can tell, there is no good way to do this today. For now we have a hand coded
mapping, but that feels very fragile as spark evolves. Is there a better way I'm missing?
And if not, could we create one? Adding a typeTag or scalaTypeTag method to DataType, or at
least to AtomicType  seems easy enough.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message