hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Vary (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-19718) Adding partitions in bulk also fetches table for each partition
Date Tue, 05 Jun 2018 10:14:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-19718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501559#comment-16501559
] 

Peter Vary commented on HIVE-19718:
-----------------------------------

The test results are the following on my laptop (~10% improvement):
* Without patch
{code}
Operation                      AMean    Mean     Med      Min      Max      Err%    
addPartitions.10               0.000    140.3    139.2    132.8    149.8    4.293 
addPartitions.100              0.000    1155     1141     1106     1218     3.456    
addPartitions.1000             0.000    10970   10940  10590   11310   2.162
{code}
* With patch
{code}
Operation                      AMean    Mean     Med      Min      Max      Err%    
addPartitions.10               0.000    133.1    132.0    125.9    143.9    4.523   
addPartitions.100              0.000    1042     1038     982.9    1125     4.519   
addPartitions.1000             0.000    9957     9948     9837     10110    0.8301 
{code}

> Adding partitions in bulk also fetches table for each partition
> ---------------------------------------------------------------
>
>                 Key: HIVE-19718
>                 URL: https://issues.apache.org/jira/browse/HIVE-19718
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore
>    Affects Versions: 3.0.0
>         Environment: Looking at {{convertToMPart}}:
> {code:Java}
>   private MPartition convertToMPart(Partition part, boolean useTableCD)
>       throws InvalidObjectException, MetaException {
>     MTable mt = getMTable(part.getCatName(), part.getDbName(), part.getTableName());
>     ...
> {code}
> So what we have as a result is that we fetch table for every partition where it should
be done just once.
>            Reporter: Alexander Kolbasov
>            Assignee: Peter Vary
>            Priority: Major
>         Attachments: HIVE-19718.2.patch, HIVE-19718.3.patch, HIVE-19718.4.patch, HIVE-19718.patch
>
>
> The ObjectStore.addPartitions() method does this:
> {code:java}
> for (Partition part : parts) {
>   if (!part.getTableName().equals(tblName) || !part.getDbName().equals(dbName)) {
>     throw new MetaException("Partition does not belong to target table "
>         + dbName + "." + tblName + ": " + part);
>   }
>   MPartition mpart = convertToMPart(part, true); // <-- Here
>   toPersist.add(mpart);
>   ...{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message