hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Vary (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20977) Lazy evaluate the table object in PreReadTableEvent to improve get_partition performance
Date Fri, 15 Feb 2019 12:39:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16769272#comment-16769272
] 

Peter Vary commented on HIVE-20977:
-----------------------------------

[~karthik.manamcheri]: The change could not be applied to master :(

Could you please rebase?

Thanks,

Peter

> Lazy evaluate the table object in PreReadTableEvent to improve get_partition performance
> ----------------------------------------------------------------------------------------
>
>                 Key: HIVE-20977
>                 URL: https://issues.apache.org/jira/browse/HIVE-20977
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Karthik Manamcheri
>            Assignee: Karthik Manamcheri
>            Priority: Minor
>             Fix For: 4.0.0
>
>         Attachments: HIVE-20977.1.patch, HIVE-20977.2.patch, HIVE-20977.3.patch, HIVE-20977.4.patch
>
>
> The PreReadTableEvent is generated for non-table operations (such as get_partitions),
but only if there is an event listener attached. However, this is also not necessary if the
event listener is not interested in the read table event.
> For example, the TransactionalValidationListener's onEvent looks like this
> {code:java}
> @Override
> public void onEvent(PreEventContext context) throws MetaException, NoSuchObjectException,
>     InvalidOperationException {
>   switch (context.getEventType()) {
>     case CREATE_TABLE:
>       handle((PreCreateTableEvent) context);
>       break;
>     case ALTER_TABLE:
>       handle((PreAlterTableEvent) context);
>       break;
>     default:
>       //no validation required..
>   }
> }{code}
>  
> Note that for read table events it is a no-op. The problem is that the get_table is evaluated
when creating the PreReadTableEvent finally to be just ignored!
> Look at the code below.. {{getMS().getTable(..)}} is evaluated irrespective of if the
listener uses it or not.
> {code:java}
> private void fireReadTablePreEvent(String catName, String dbName, String tblName)
>     throws MetaException, NoSuchObjectException {
>   if(preListeners.size() > 0) {
>     // do this only if there is a pre event listener registered (avoid unnecessary
>     // metastore api call)
>     Table t = getMS().getTable(catName, dbName, tblName);
>     if (t == null) {
>       throw new NoSuchObjectException(TableName.getQualified(catName, dbName, tblName)
>           + " table not found");
>     }
>     firePreEvent(new PreReadTableEvent(t, this));
>   }
> }
> {code}
> This can be improved by using a {{Supplier}} and lazily evaluating the table when needed
(once when the first time it is called, memorized after that).
> *Motivation*
> Whenever a partition call occurs (get_partition, etc.), we fire the PreReadTableEvent.
This affects performance since it fetches the table even if it is not being used. This change
will improve performance on the get_partition calls.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message