hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth Jayachandran (Jira)" <j...@apache.org>
Subject [jira] [Resolved] (HIVE-24020) Automatic Compaction not working in existing partitions for Streaming Ingest with Dynamic Partition
Date Wed, 26 Aug 2020 17:52:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-24020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Prasanth Jayachandran resolved HIVE-24020.
------------------------------------------
    Fix Version/s: 4.0.0
       Resolution: Fixed

> Automatic Compaction not working in existing partitions for Streaming Ingest with Dynamic
Partition
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-24020
>                 URL: https://issues.apache.org/jira/browse/HIVE-24020
>             Project: Hive
>          Issue Type: Bug
>          Components: Streaming, Transactions
>    Affects Versions: 4.0.0, 3.1.2
>            Reporter: Vipin Vishvkarma
>            Assignee: Vipin Vishvkarma
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.0.0
>
>          Time Spent: 1h
>  Remaining Estimate: 0h
>
> This issue happens when we try to do streaming ingest with dynamic partition on already
existing partitions. I checked in the code, we have following check in the AbstractRecordWriter.
>  
> {code:java}
> PartitionInfo partitionInfo = conn.createPartitionIfNotExists(partitionValues);
> // collect the newly added partitions. connection.commitTransaction() will report the
dynamically added
> // partitions to TxnHandler
> if (!partitionInfo.isExists()) {
>   addedPartitions.add(partitionInfo.getName());
> } else {
>   if (LOG.isDebugEnabled()) {
>     LOG.debug("Partition {} already exists for table {}",
>         partitionInfo.getName(), fullyQualifiedTableName);
>   }
> }
> {code}
> Above *addedPartitions* is passed to *addDynamicPartitions* during TransactionBatch
commit. So in case of already existing partitions, *addedPartitions* will be empty and *addDynamicPartitions* **will
not move entries from TXN_COMPONENTS to COMPLETED_TXN_COMPONENTS. This results in Initiator
not able to trigger auto compaction.
> Another issue which has been observed is, we are not clearing *addedPartitions* on
writer close, which results in information flowing across transactions.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message