hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Work logged] (HIVE-23956) Delete delta directory file information should be pushed to execution side
Date Mon, 03 Aug 2020 09:48:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-23956?focusedWorklogId=465615&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-465615
]

ASF GitHub Bot logged work on HIVE-23956:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 03/Aug/20 09:47
            Start Date: 03/Aug/20 09:47
    Worklog Time Spent: 10m 
      Work Description: pvary commented on a change in pull request #1339:
URL: https://github.com/apache/hive/pull/1339#discussion_r464309265



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
##########
@@ -1574,20 +1577,23 @@ public int compareTo(CompressedOwid other) {
       this.orcSplit = orcSplit;
 
       try {
-        final Path[] deleteDeltaDirs = getDeleteDeltaDirsFromSplit(orcSplit);
-        if (deleteDeltaDirs.length > 0) {
+        if (orcSplit.getDeltas().size() > 0) {
           AcidOutputFormat.Options orcSplitMinMaxWriteIds =
               AcidUtils.parseBaseOrDeltaBucketFilename(orcSplit.getPath(), conf);
           int totalDeleteEventCount = 0;
-          for (Path deleteDeltaDir : deleteDeltaDirs) {
-            if (!isQualifiedDeleteDeltaForSplit(orcSplitMinMaxWriteIds, deleteDeltaDir))
{
-              continue;
-            }
-            Path[] deleteDeltaFiles = OrcRawRecordMerger.getDeltaFiles(deleteDeltaDir, bucket,
-                new OrcRawRecordMerger.Options().isCompacting(false), null);
-            for (Path deleteDeltaFile : deleteDeltaFiles) {
-              try {
-                ReaderData readerData = getOrcTail(deleteDeltaFile, conf, cacheTag);
+          for (AcidInputFormat.DeltaMetaData deltaMetaData : orcSplit.getDeltas()) {
+            // We got one path for each statement in a multiStmt transaction
+            for (Pair<Path,Integer> deleteDeltaDir : deltaMetaData.getPaths(orcSplit.getRootDir()))
{

Review comment:
       nit: space <Path, Integer>




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 465615)
    Time Spent: 3h 20m  (was: 3h 10m)

> Delete delta directory file information should be pushed to execution side
> --------------------------------------------------------------------------
>
>                 Key: HIVE-23956
>                 URL: https://issues.apache.org/jira/browse/HIVE-23956
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Peter Varga
>            Assignee: Peter Varga
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Since HIVE-23840 LLAP cache is used to retrieve the tail of the ORC bucket files in the
delete deltas, but to use the cache the fileId must be determined, so one more FileSystem
call is issued for each bucket.
> This fileId is already available during compilation in the AcidState calculation, we
should serialise this to the OrcSplit, and remove the unnecessary FS calls.
> Furthermore instead of sending the SyntheticFileId directly, we should pass the attemptId
instead of the standard path hash, this way the path and the SyntheticFileId. can be calculated,
and it will work even, if the move free delete operations will be introduced.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message