hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <>
Subject [jira] [Work logged] (HIVE-24020) Automatic Compaction not working in existing partitions for Streaming Ingest with Dynamic Partition
Date Wed, 26 Aug 2020 17:22:00 GMT


ASF GitHub Bot logged work on HIVE-24020:

                Author: ASF GitHub Bot
            Created on: 26/Aug/20 17:21
            Start Date: 26/Aug/20 17:21
    Worklog Time Spent: 10m 
      Work Description: vpnvishv commented on pull request #1382:

   @pvary @prasanthj  Can you please merge this PR.

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:

Issue Time Tracking

    Worklog Id:     (was: 474904)
    Time Spent: 50m  (was: 40m)

> Automatic Compaction not working in existing partitions for Streaming Ingest with Dynamic
> ---------------------------------------------------------------------------------------------------
>                 Key: HIVE-24020
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Streaming, Transactions
>    Affects Versions: 4.0.0, 3.1.2
>            Reporter: Vipin Vishvkarma
>            Assignee: Vipin Vishvkarma
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 50m
>  Remaining Estimate: 0h
> This issue happens when we try to do streaming ingest with dynamic partition on already
existing partitions. I checked in the code, we have following check in the AbstractRecordWriter.
> {code:java}
> PartitionInfo partitionInfo = conn.createPartitionIfNotExists(partitionValues);
> // collect the newly added partitions. connection.commitTransaction() will report the
dynamically added
> // partitions to TxnHandler
> if (!partitionInfo.isExists()) {
>   addedPartitions.add(partitionInfo.getName());
> } else {
>   if (LOG.isDebugEnabled()) {
>     LOG.debug("Partition {} already exists for table {}",
>         partitionInfo.getName(), fullyQualifiedTableName);
>   }
> }
> {code}
> Above *addedPartitions* is passed to *addDynamicPartitions* during TransactionBatch
commit. So in case of already existing partitions, *addedPartitions* will be empty and *addDynamicPartitions* **will
not move entries from TXN_COMPONENTS to COMPLETED_TXN_COMPONENTS. This results in Initiator
not able to trigger auto compaction.
> Another issue which has been observed is, we are not clearing *addedPartitions* on
writer close, which results in information flowing across transactions.

This message was sent by Atlassian Jira

View raw message