hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergio Peña (JIRA) <>
Subject [jira] [Commented] (HIVE-13884) Disallow queries in HMS fetching more than a configured number of partitions
Date Sat, 02 Jul 2016 16:33:10 GMT


Sergio Peña commented on HIVE-13884:

[~leftylev] Here's the description of the new parameter:

This limits the number of partitions that can be requested from the metastore for a given
table. A query will not be executed if it attempts to fetch more partitions per table than
the limit configured. This parameter is preferred over hive.limit.query.max.table.partition
(deprecated). The default value is -1 (unlimited). 

> Disallow queries in HMS fetching more than a configured number of partitions
> ----------------------------------------------------------------------------
>                 Key: HIVE-13884
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Mohit Sabharwal
>            Assignee: Sergio Peña
>              Labels: TODOC2.2
>             Fix For: 2.2.0
>         Attachments: HIVE-13884.1.patch, HIVE-13884.10.patch, HIVE-13884.2.patch, HIVE-13884.3.patch,
HIVE-13884.4.patch, HIVE-13884.5.patch, HIVE-13884.6.patch, HIVE-13884.7.patch, HIVE-13884.8.patch,
> Currently the PartitionPruner requests either all partitions or partitions based on filter
expression. In either scenarios, if the number of partitions accessed is large there can be
significant memory pressure at the HMS server end.
> We already have a config {{hive.limit.query.max.table.partition}} that enforces limits
on number of partitions that may be scanned per operator. But this check happens after the
PartitionPruner has already fetched all partitions.
> We should add an option at PartitionPruner level to disallow queries that attempt to
access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition filter in
PartitionPruner, but this check accepts any query with a pruning condition, even if partitions
fetched are large. In multi-tenant environments, admins could use more control w.r.t. number
of partitions allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names (instead of partition
specs) and throw an exception if number of partitions exceeds the configured value. Otherwise,
fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended to take
partition filter expressions like {{getPartitionsByExpr}} call does.

This message was sent by Atlassian JIRA

View raw message