hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-18781) Create/Replicate Open, Commit (without writes) and Abort Txn events
Date Tue, 03 Apr 2018 07:56:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-18781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423607#comment-16423607
] 

ASF GitHub Bot commented on HIVE-18781:
---------------------------------------

Github user maheshk114 closed the pull request at:

    https://github.com/apache/hive/pull/311


> Create/Replicate Open, Commit (without writes) and Abort Txn events
> -------------------------------------------------------------------
>
>                 Key: HIVE-18781
>                 URL: https://issues.apache.org/jira/browse/HIVE-18781
>             Project: Hive
>          Issue Type: Sub-task
>          Components: repl, Transactions
>    Affects Versions: 3.0.0
>            Reporter: mahesh kumar behera
>            Assignee: mahesh kumar behera
>            Priority: Major
>              Labels: ACID, DR, pull-request-available, replication
>             Fix For: 3.0.0
>
>         Attachments: HIVE-18781.01.patch, HIVE-18781.02.patch, HIVE-18781.03.patch, HIVE-18781.04.patch,
HIVE-18781.05.patch, HIVE-18781.06.patch, HIVE-18781.07.patch, HIVE-18781.08.patch, HIVE-18781.09.patch,
HIVE-18781.10.patch, HIVE-18781.11.patch, HIVE-18781.12.patch
>
>
> *EVENT_OPEN_TXN:*
>  *Source Warehouse:*
>  - Create new event type EVENT_OPEN_TXN with related message format etc.
>  - When any transaction is opened either by auto-commit mode or multi-statement mode,
need to capture this event.
>  - Repl dump should read this event from EventNotificationTable and dump the message.
> *Target Warehouse:*
>  - Repl load should read the event from the dump and get the message.
>  - Open a txn in target warehouse.
>  - Create a map of source txn ID against target txn ID and persist the same in metastore.
There should be one map per replication policy (DBName.* incase of DB level replication, DBName.TableName
incase of table level replication)
>  
> *EVENT_COMMIT_TXN (Without writes)*
> Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions modified within
the txn.
> *Source warehouse:*
>  - Create EVENT_COMMIT_TXN event type with corresponding message format etc.
> *Target warehouse:*
>  - Repl load should read this event from the dump.
>  - Validate the source txn ID from the event using the Source-Target Txn ID map maintained
in target metastore. Also, need to check if corresponding target txn ID is valid.
>  - If valid, then apply the event and commit the corresponding target transaction.
>  - This new event should be idempotent such that if it is applied twice, then second
time it should be loop.
>  
> *EVENT_ABORT_TXN*
>  Source Warehouse:
>  - Create new event type EVENT_ABORT_TXN with related message format etc.
>  - Capture this event when abort the txn.
>  - Repl dump should read this event from EventNotificationTable and dump the message.
> *Target Warehouse:*
>  - Repl load should read the event from the dump and get the message.
>  - Validate if source txn ID from the event is there in the source-target txn ID map.
If not there, just noop the event.
>  - If valid, then Abort the corresponding target txn and remove the entry from source-target
txn ID map.
> All these new events should be idempotent such that if it is applied twice, then second
time it should be noop.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message