hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <>
Subject [jira] [Commented] (HIVE-17466) Metastore API to list unique partition-key-value combinations
Date Fri, 08 Sep 2017 21:37:00 GMT


Alan Gates commented on HIVE-17466:

I'm certainly ok with the intention.

At some point I'd like to rationalize the thrift API, as it has many many calls.  But we should
do that holistically rather than start blocking individual changes.

> Metastore API to list unique partition-key-value combinations
> -------------------------------------------------------------
>                 Key: HIVE-17466
>                 URL:
>             Project: Hive
>          Issue Type: New Feature
>          Components: Metastore
>    Affects Versions: 2.2.0, 3.0.0
>            Reporter: Mithun Radhakrishnan
>            Assignee: Thiruvel Thirumoolan
>         Attachments: HIVE-17466.1.patch, HIVE-17466.2-branch-2.patch, HIVE-17466.2.patch
> Raising this on behalf of [~thiruvel], who wrote this initially as part of a tangential
"data-discovery" system.
> Programs like Apache Oozie, Apache Falcon (or Yahoo GDM), etc. launch workflows based
on the availability of table/partitions. Partitions are currently discovered by listing partitions
using (what boils down to) {{HiveMetaStoreClient.listPartitions()}}. This can be slow and
cumbersome, given that {{Partition}} objects are heavyweight and carry redundant information.
The alternative is to use partition-names, which will need client-side parsing to extract
part-key values.
> When checking which hourly partitions for a particular day have been published already,
it would be preferable to have an API that pushed down part-key extraction into the {{RawStore}}
layer, and returned key-values as the result. This would be similar to how {{SELECT DISTINCT
part_key FROM my_table;}} would run, but at the {{HiveMetaStoreClient}} level.
> Here's what we've been using at Yahoo.

This message was sent by Atlassian JIRA

View raw message