spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject [SQL] Why casting string column to timestamp gives null?
Date Fri, 07 Jun 2019 16:38:43 GMT
Hi,

Why is casting a string column to timestamp not giving the same results as
going through casting to long in-between? I'm tempted to consider it a bug.

scala> spark.version
res4: String = 2.4.3

scala> Seq("1", "2").toDF("ts").select($"ts" cast "timestamp").show
+----+
|  ts|
+----+
|null|
|null|
+----+

scala> Seq("1", "2").toDF("ts").select($"ts" cast "long").select($"ts" cast
"timestamp").show
+-------------------+
|                 ts|
+-------------------+
|1970-01-01 01:00:01|
|1970-01-01 01:00:02|
+-------------------+

Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
The Internals of Spark SQL https://bit.ly/spark-sql-internals
The Internals of Spark Structured Streaming
https://bit.ly/spark-structured-streaming
The Internals of Apache Kafka https://bit.ly/apache-kafka-internals
Follow me at https://twitter.com/jaceklaskowski

Mime
View raw message