Hi
I see a behaviour change after testing spark2. Timestamp with milliseconds
(from java for instance) can no longer be processed using TimestampType
while reading a json.
I guess this is due to the following code:
https://github.com/apache/spark/pull/10687/commits/600f73f385347d4a2ed8ca21f3268c01cd8855f7
Is it an expected behaviour, a bug?
Alternative 1 is to parse it as LongType and then loose the milliseconds:
cast (logEntrydate/1000 as TimeStamp)
or
Dataset<Timestamp> ds2 = ds1.map(row->{
long l = row.getLong(1);
return new Timestamp(l);
}, Encoders.TIMESTAMP());
Is there another better way to do it?
Thanks!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/json-with-millisecond-timestamp-in-spark-2-tp27614.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
|