hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt McCline (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11981) ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)
Date Thu, 19 Nov 2015 10:53:11 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15013322#comment-15013322
] 

Matt McCline commented on HIVE-11981:
-------------------------------------

The new parameter  hive.exec.schema.evolution is intended to be general and not be ORC only.
 Other file formats that contain schema metadata (e.g. parquet) could add the marker interface
SelfDescribingInputFormatInterface and provide schema evolution functionality.

> ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)
> ------------------------------------------------------------------
>
>                 Key: HIVE-11981
>                 URL: https://issues.apache.org/jira/browse/HIVE-11981
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive, Transactions
>            Reporter: Matt McCline
>            Assignee: Matt McCline
>            Priority: Critical
>              Labels: TODOC2.0
>             Fix For: 2.0.0
>
>         Attachments: HIVE-11981.01.patch, HIVE-11981.02.patch, HIVE-11981.03.patch, HIVE-11981.05.patch,
HIVE-11981.06.patch, HIVE-11981.07.patch, HIVE-11981.08.patch, HIVE-11981.09.patch, HIVE-11981.091.patch,
HIVE-11981.092.patch, HIVE-11981.093.patch, HIVE-11981.094.patch, HIVE-11981.095.patch, HIVE-11981.096.patch,
HIVE-11981.097.patch, HIVE-11981.098.patch, HIVE-11981.099.patch, HIVE-11981.0991.patch, HIVE-11981.0992.patch,
ORC Schema Evolution Issues.docx
>
>
> High priority issues with schema evolution for the ORC file format.
> Schema evolution here is limited to adding new columns and a few cases of column type-widening
(e.g. int to bigint).
> Renaming columns, deleting column, moving columns and other schema evolution were not
pursued due to lack of importance and lack of time.  Also, it appears a much more sophisticated
metadata would be needed to support them.
> The biggest issues for users have been adding new columns for ACID table (HIVE-11421
Support Schema evolution for ACID tables) and vectorization (HIVE-10598 Vectorization borks
when column is added to table).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message