hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abhishek Somani (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
Date Tue, 12 May 2020 17:23:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105622#comment-17105622
] 

Abhishek Somani commented on HIVE-21052:
----------------------------------------

 This seems like a critical issue. Is this actively being worked on? This affects both Hive
3 and current master (Hive 4), right?

cc  [~pvary] [~gopalv] 

> Make sure transactions get cleaned if they are aborted before addPartitions is called
> -------------------------------------------------------------------------------------
>
>                 Key: HIVE-21052
>                 URL: https://issues.apache.org/jira/browse/HIVE-21052
>             Project: Hive
>          Issue Type: Bug
>          Components: Transactions
>    Affects Versions: 3.0.0, 3.1.1
>            Reporter: Jaume M
>            Assignee: Jaume M
>            Priority: Critical
>         Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, HIVE-21052.10.patch,
HIVE-21052.11.patch, HIVE-21052.12.patch, HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch,
HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch, HIVE-21052.8.patch, HIVE-21052.9.patch
>
>
> If the transaction is aborted between openTxn and addPartitions and data has been written
on the table the transaction manager will think it's an empty transaction and no cleaning
will be done.
> This is currently an issue in the streaming API and in micromanaged tables. As proposed
by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and when addPartitions
is called remove this entry from TXN_COMPONENTS and add the corresponding partition entry
to TXN_COMPONENTS.
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that specifies
that a transaction was opened and it was aborted it must generate jobs for the worker for
every possible partition available.
> cc [~ewohlstadter]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message