hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth Jayachandran (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HIVE-20707) Automatic partition management
Date Fri, 12 Oct 2018 01:29:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16647268#comment-16647268
] 

Prasanth Jayachandran edited comment on HIVE-20707 at 10/12/18 1:28 AM:
------------------------------------------------------------------------

There is more scope added to this patch

1) Support managed and external tables. Managed tables will get X lock for MSCK REPAIR.

2) Retention period for partitions can be specified via table property.

3) MSCK repair can now handle retention periods via config.

4) All external tables will now automatically enable partition discovery.

On top of these there are fixes related to ACID, MSCK was acquiring wrong lock, explain locks
was not working for msck, fixes related to catalog handling (some metastore APIs was not handling
catalog names correctly).


was (Author: prasanth_j):
There is more scope added to this patch

1) Support managed and external tables. Managed tables will get X lock for MSCK.

2) Retention period for partitions can be specified via table property.

3) MSCK repair can now handle retention periods via config.

4) All external tables will now automatically enable partition discovery.

On top of these there are fixes related to ACID, MSCK was acquiring wrong lock, explain locks
was not working for msck, fixes related to catalog handling (some metastore APIs was not handling
catalog names correctly).

> Automatic partition management
> ------------------------------
>
>                 Key: HIVE-20707
>                 URL: https://issues.apache.org/jira/browse/HIVE-20707
>             Project: Hive
>          Issue Type: New Feature
>    Affects Versions: 4.0.0
>            Reporter: Prasanth Jayachandran
>            Assignee: Prasanth Jayachandran
>            Priority: Major
>         Attachments: HIVE-20707.1.patch
>
>
> In current scenario, to add partitions for external tables to metastore, MSCK REPAIR
command has to be executed manually. To avoid this manual step, external tables can be specified
a table property based on which a background metastore thread can sync partitions periodically.
Tables can also be specified with partition retention period. Any partition whose age exceeds
the retention period will be dropped automatically.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message