hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HIVE-19124) implement a basic major compactor for MM tables
Date Fri, 06 Apr 2018 21:32:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429030#comment-16429030
] 

Sergey Shelukhin edited comment on HIVE-19124 at 4/6/18 9:31 PM:
-----------------------------------------------------------------

Fixing initiator to not skip MM tables, addressing most of the other feedback.
Also making sure that compactor doesn't commit a new txn that writes to the table, instead
just like a regular compactor it commits by moving the directory into place.

One thing that is not addressed is the parallel writer case, which is the same issue as HIVE-18570.
Let me see if I can address it for compaction specifically...

cc [~gopalv]
Also looking at the Cleaner it seems like it should run just fine for MM tables and delete
obsolete directories. It never checks table type... [~ekoifman] are you sure there's a problem
with it?


was (Author: sershe):
Fixing initiator to not skip MM tables, addressing most of the other feedback.
Also making sure that compactor doesn't commit a new txn that writes to the table, instead
just like a regular compactor it commits by moving the directory into place.

One thing that is not addressed is the parallel writer case, which is the same issue as HIVE-18570.
Let me see if I can address it for compaction specifically...

Also looking at the Cleaner it seems like it should run just fine for MM tables and delete
obsolete directories. It never checks table type... [~ekoifman] are you sure there's a problem
with it?

> implement a basic major compactor for MM tables
> -----------------------------------------------
>
>                 Key: HIVE-19124
>                 URL: https://issues.apache.org/jira/browse/HIVE-19124
>             Project: Hive
>          Issue Type: Bug
>          Components: Transactions
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>            Priority: Major
>              Labels: mm-gap-2
>         Attachments: HIVE-19124.01.patch, HIVE-19124.patch
>
>
> For now, it will run a query directly and only major compactions will be supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message