hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject [jira] [Commented] (HIVE-21354) Lock The Entire Table If Majority Of Partitions Are Locked
Date Fri, 01 Mar 2019 01:19:00 GMT


BELUGA BEHR commented on HIVE-21354:

Thanks for the input [~gopalv].

What about just a simple {{SELECT * FROM TABLE WHERE (non-partitioned-value)=?}}

> Lock The Entire Table If Majority Of Partitions Are Locked
> ----------------------------------------------------------
>                 Key: HIVE-21354
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>          Components: HiveServer2
>    Affects Versions: 4.0.0, 3.2.0
>            Reporter: BELUGA BEHR
>            Priority: Major
> One of the bottlenecks of any Hive query is the ZooKeeper locking mechanism.  When a
Hive query interacts with a table which has a lot of partitions, this may put a lot of stress
on the ZK system.
> Please add a heuristic that works like this:
> # Count the number of partitions that a query is required to lock
> # Obtain the total number of partitions in the table
> # If the number of partitions accessed by the query is greater than or equal to half
the total number of partitions, simply create one ZNode lock at the table level.
> This would improve performance of many queries, but in particular, a {{select count(1)
from table}} ... or ... {{select * from table limit 5}} where the table has many partitions.

This message was sent by Atlassian JIRA

View raw message