spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From filousen <filou...@hotmail.com>
Subject json with millisecond timestamp in spark 2
Date Mon, 29 Aug 2016 07:05:56 GMT
Hi

I see a behaviour change after testing spark2. Timestamp with milliseconds
(from java for instance) can no longer be processed using TimestampType
while reading a json.
I guess this is due to the following code:
https://github.com/apache/spark/pull/10687/commits/600f73f385347d4a2ed8ca21f3268c01cd8855f7

Is it an expected behaviour, a bug? 

Alternative 1 is to parse it as LongType and then loose the milliseconds:
cast (logEntrydate/1000 as TimeStamp)
or 
Dataset<Timestamp> ds2 = ds1.map(row->{
            long l = row.getLong(1);
            return new Timestamp(l);
            
        }, Encoders.TIMESTAMP());

Is there another better way to do it?

Thanks!



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/json-with-millisecond-timestamp-in-spark-2-tp27614.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message