spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From filousen <>
Subject json with millisecond timestamp in spark 2
Date Mon, 29 Aug 2016 07:05:56 GMT

I see a behaviour change after testing spark2. Timestamp with milliseconds
(from java for instance) can no longer be processed using TimestampType
while reading a json.
I guess this is due to the following code:

Is it an expected behaviour, a bug? 

Alternative 1 is to parse it as LongType and then loose the milliseconds:
cast (logEntrydate/1000 as TimeStamp)
Dataset<Timestamp> ds2 =>{
            long l = row.getLong(1);
            return new Timestamp(l);
        }, Encoders.TIMESTAMP());

Is there another better way to do it?


View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe e-mail:

View raw message