spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michal Vince <vince.mic...@gmail.com>
Subject 1.6.0 spark.sql datetime conversion problem
Date Fri, 04 Mar 2016 17:08:19 GMT
Hi guys
I`m using spark 1.6.0 and I`m not sure if I found a bug or I`m doing 
something wrong

I`m playing with dataframes and I`m converting iso 8601 with millis to 
my timezone - which is Europe/Bratislava with fromt_utc_timestamp 
function from spark.sql.functions

the problem is that Europe/Bratislava is UTC+1 hour in february, and 
from 27th March it`s going to be UTC+2h but from_utc_timestamp ignores 
this and always converts to UTC+2h

am doing something wrong when converting?


e.g.
myDF.withColumn("time", 
from_utc_timestamp(l.col("timestamp"),"Europe/Bratislava"))

and the output:
time 	timestamp


	
2016-02-22 02:59:11.0 	2016-02-22T00:59:11.000Z
2016-02-20 20:16:35.0 	2016-02-20T18:16:35.000Z
2016-02-20 05:17:29.0 	2016-02-20T03:17:29.000Z
2016-02-18 18:29:06.0 	2016-02-18T16:29:06.000Z
2016-02-17 01:47:20.0 	2016-02-16T23:47:20.000Z
2016-02-15 07:05:04.0 	2016-02-15T05:05:04.000Z
2016-02-13 23:25:14.0 	2016-02-13T21:25:14.000Z



Mime
View raw message