spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jestin Ma <jestinwith.a...@gmail.com>
Subject DataFramesWriter saving DataFrames timestamp in weird format
Date Thu, 11 Aug 2016 21:04:30 GMT
When I load in a timestamp column and try to save it immediately without
any transformations, the output time is unix time with padded 0's until
there are 16 values.

For example,
loading in a time of August 3, 2016, 00:36:25 GMT, which is 1470184585 in
UNIX time, saves as 1470184585000000.

When I do df.show(), it shows the date format that I pass in (custom
format), but it saves as I mentioned.
I tried loading the saved file as a timestamp and it expectedly throws an
exception, not being able to recognize an invalid time.

Are there any explanations / workarounds for this?

Thank you,
Jestin

Mime
View raw message