hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "mahesh kumar behera (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-19089) Create/Replicate Allocate write-id event
Date Mon, 02 Apr 2018 10:33:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-19089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

mahesh kumar behera updated HIVE-19089:
---------------------------------------
    Description: 
*EVENT_ALLOCATE_WRITE_ID*
*Source Warehouse:*
 * Create new event type EVENT_ALLOCATE_WRITE_ID with related message format etc.

 * Capture this event when allocate a table write ID from the sequence table by ACID operation.

 * Repl dump should read this event from EventNotificationTable and dump the message.

*Target Warehouse:*
 * Repl load should read the event from the dump and get the message.

 * Validate if source txn ID from the event is there in the source-target txn ID map. If not
there, just noop the event.

 * If valid, then Allocate table write ID from sequence table

*Extend listener notify event API to add two new parameter , dbconn and sqlgenerator to add the
events to notification_log table within the same transaction* 

  was:
*EVENT_OPEN_TXN:*
 *Source Warehouse:*
 - Create new event type EVENT_OPEN_TXN with related message format etc.
 - When any transaction is opened either by auto-commit mode or multi-statement mode, need
to capture this event.
 - Repl dump should read this event from EventNotificationTable and dump the message.

*Target Warehouse:*
 - Repl load should read the event from the dump and get the message.
 - Open a txn in target warehouse.
 - Create a map of source txn ID against target txn ID and persist the same in metastore.
There should be one map per replication policy (DBName.* incase of DB level replication, DBName.TableName
incase of table level replication)

 

*EVENT_COMMIT_TXN (Without writes)*

Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions modified within
the txn.

*Source warehouse:*
 - Create EVENT_COMMIT_TXN event type with corresponding message format etc.

*Target warehouse:*
 - Repl load should read this event from the dump.
 - Validate the source txn ID from the event using the Source-Target Txn ID map maintained
in target metastore. Also, need to check if corresponding target txn ID is valid.
 - If valid, then apply the event and commit the corresponding target transaction.
 - This new event should be idempotent such that if it is applied twice, then second time
it should be loop.

 

*EVENT_ABORT_TXN*
 Source Warehouse:
 - Create new event type EVENT_ABORT_TXN with related message format etc.
 - Capture this event when abort the txn.
 - Repl dump should read this event from EventNotificationTable and dump the message.

*Target Warehouse:*
 - Repl load should read the event from the dump and get the message.
 - Validate if source txn ID from the event is there in the source-target txn ID map. If not
there, just noop the event.
 - If valid, then Abort the corresponding target txn and remove the entry from source-target
txn ID map.

All these new events should be idempotent such that if it is applied twice, then second time
it should be noop.


> Create/Replicate Allocate write-id event
> ----------------------------------------
>
>                 Key: HIVE-19089
>                 URL: https://issues.apache.org/jira/browse/HIVE-19089
>             Project: Hive
>          Issue Type: Sub-task
>          Components: repl, Transactions
>    Affects Versions: 3.0.0
>            Reporter: mahesh kumar behera
>            Assignee: mahesh kumar behera
>            Priority: Major
>              Labels: ACID, DR, replication
>             Fix For: 3.0.0
>
>
> *EVENT_ALLOCATE_WRITE_ID*
> *Source Warehouse:*
>  * Create new event type EVENT_ALLOCATE_WRITE_ID with related message format etc.
>  * Capture this event when allocate a table write ID from the sequence table by ACID
operation.
>  * Repl dump should read this event from EventNotificationTable and dump the message.
> *Target Warehouse:*
>  * Repl load should read the event from the dump and get the message.
>  * Validate if source txn ID from the event is there in the source-target txn ID map.
If not there, just noop the event.
>  * If valid, then Allocate table write ID from sequence table
> *Extend listener notify event API to add two new parameter , dbconn and sqlgenerator
to add the events to notification_log table within the same transaction* 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message