spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ZHANG Wei <wezh...@outlook.com>
Subject Re: Spark ORC store written timestamp as column
Date Fri, 24 Apr 2020 11:12:51 GMT
>From what I think I understand, the OrcOutputWriter leverages orc-core
to write. I'm wondering if ORC supports the row metadata or not. If
not, maybe the org.apache.orc.Writer::addRowBatch() can be overrided to
record the metadata after RowBatch written.

-- 
Cheers,
-z

On Thu, 16 Apr 2020 04:47:31 +0000
Manjunath Shetty H <manjunathshetty@live.com> wrote:

> Hi All,
> 
> Is there anyway to store the exact written timestamp in the ORC file through spark ?.
> Use case something like `current_timestamp()` function in SQL. Generating in the program
will not be equal to actual write time in ORC/hdfs file.
> 
> Any suggestions will be helpful.
> 
> 
> Thanks
> Manjunath

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message