spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Murali Aakula (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-26994) Enhance StructField to accept number format or date format
Date Thu, 28 Feb 2019 14:31:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-26994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16780572#comment-16780572
] 

Murali Aakula commented on SPARK-26994:
---------------------------------------

Sample Input:

id,cpt,schedule_date,dstamp,weight,miles
1,MON-14:30,2018-11-30 10:00,29/11/2018 21:07:47,"10,110.12","1,010.12"
2,MON-14:30,2018-11-30 11:00,29/11/2018 21:17:47,"1,110.12","10.10"

Currently, we are reading few columns as string(schedule_date, dstamp, weight, miles) then
converting them using spark sql functions as needed(date, date, double, double). Kind of Two
step process. We can set one date and one timestamp format on spark config, but not multiple.

> Enhance StructField to accept number format or date format
> ----------------------------------------------------------
>
>                 Key: SPARK-26994
>                 URL: https://issues.apache.org/jira/browse/SPARK-26994
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core, SQL
>    Affects Versions: 2.4.0
>            Reporter: Murali Aakula
>            Priority: Major
>
> Enhance StructField to accept number format or date format and Enahcne reader/steramreader
and writer/streamwriter to use these formats.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message