spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anand Viswanathan <anand_v...@ymail.com.INVALID>
Subject Re: spark infers date to be timestamp type
Date Thu, 27 Oct 2016 00:12:52 GMT
Hi,

you can use the customSchema(for DateType) and specify dateFormat in .option().
or 
at spark dataframe side, you can convert the timestamp to date using cast to the column.

Thanks and regards,
Anand Viswanathan

> On Oct 26, 2016, at 8:07 PM, Koert Kuipers <koert@tresata.com> wrote:
> 
> hey,
> i create a file called test.csv with contents:
> date
> 2015-01-01
> 2016-03-05
> 
> next i run this code in spark 2.0.1:
> spark.read
>   .format("csv")
>   .option("header", true)
>   .option("inferSchema", true)
>   .load("test.csv")
>   .printSchema
> 
> the result is:
> root
>  |-- date: timestamp (nullable = true)
> 
> 
> On Wed, Oct 26, 2016 at 7:35 PM, Hyukjin Kwon <gurwls223@gmail.com <mailto:gurwls223@gmail.com>>
wrote:
> There are now timestampFormat for TimestampType and dateFormat for DateType.
> 
> Do you mind if I ask to share your codes?
> 
> 
> On 27 Oct 2016 2:16 a.m., "Koert Kuipers" <koert@tresata.com <mailto:koert@tresata.com>>
wrote:
> is there a reason a column with dates in format yyyy-mm-dd in a csv file is inferred
to be TimestampType and not DateType?
> 
> thanks! koert
> 


Mime
View raw message