hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (HIVE-21529) Hive support bootstrap of ACID/MM tables on an existing policy.
Date Mon, 15 Apr 2019 07:25:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-21529?focusedWorklogId=227490&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227490
]

ASF GitHub Bot logged work on HIVE-21529:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 15/Apr/19 07:24
            Start Date: 15/Apr/19 07:24
    Worklog Time Spent: 10m 
      Work Description: ashutosh-bapat commented on pull request #581: HIVE-21529 : Bootstrap
ACID tables as part of incremental dump.
URL: https://github.com/apache/hive/pull/581#discussion_r275233040
 
 

 ##########
 File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
 ##########
 @@ -397,22 +469,13 @@ private String getValidWriteIdList(String dbName, String tblName, String
validTx
     return openTxns;
   }
 
-  String getValidTxnListForReplDump(Hive hiveDb) throws HiveException {
-    // Key design point for REPL DUMP is to not have any txns older than current txn in which
dump runs.
-    // This is needed to ensure that Repl dump doesn't copy any data files written by any
open txns
-    // mainly for streaming ingest case where one delta file shall have data from committed/aborted/open
txns.
-    // It may also have data inconsistency if the on-going txns doesn't have corresponding
open/write
-    // events captured which means, catch-up incremental phase won't be able to replicate
those txns.
-    // So, the logic is to wait for configured amount of time to see if all open txns <
current txn is
-    // getting aborted/committed. If not, then we forcefully abort those txns just like AcidHouseKeeperService.
-    ValidTxnList validTxnList = getTxnMgr().getValidTxns();
-    long timeoutInMs = HiveConf.getTimeVar(conf,
-            HiveConf.ConfVars.REPL_BOOTSTRAP_DUMP_OPEN_TXN_TIMEOUT, TimeUnit.MILLISECONDS);
-    long waitUntilTime = System.currentTimeMillis() + timeoutInMs;
+  ValidTxnList getValidTxnListForReplDump(Hive hiveDb, ValidTxnList validTxnList,
 
 Review comment:
   Done.
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 227490)
    Time Spent: 5h 40m  (was: 5.5h)

> Hive support bootstrap of ACID/MM tables on an existing policy.
> ---------------------------------------------------------------
>
>                 Key: HIVE-21529
>                 URL: https://issues.apache.org/jira/browse/HIVE-21529
>             Project: Hive
>          Issue Type: Sub-task
>          Components: repl, Transactions
>    Affects Versions: 4.0.0
>            Reporter: Sankar Hariappan
>            Assignee: Ashutosh Bapat
>            Priority: Major
>              Labels: DR, pull-request-available, replication
>         Attachments: HIVE-21529.01.patch, HIVE-21529.02.patch, HIVE-21529.03.patch
>
>          Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> If ACID/MM tables to be enabled (hive.repl.dump.include.acid.tables) on an existing repl
policy, then need to combine bootstrap dump of these tables along with the ongoing incremental
dump. 
>  Shall add a one time config "hive.repl.bootstrap.acid.tables" to include bootstrap in
the given dump.
> TheĀ support for hive.repl.bootstrap.cleanup.type for ACID tables to clean-up partially
bootstrapped tables in case of retry is already in place, thanks to the work done during external
tables. Need to test that it actually works.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message