hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <>
Subject [jira] [Commented] (HIVE-11229) Mutation API: Coordinator communication with meta store should be optional
Date Tue, 21 Jul 2015 17:44:04 GMT


Alan Gates commented on HIVE-11229:


> Mutation API: Coordinator communication with meta store should be optional
> --------------------------------------------------------------------------
>                 Key: HIVE-11229
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: HCatalog
>    Affects Versions: 2.0.0
>            Reporter: Elliot West
>            Assignee: Elliot West
>              Labels: streaming_api
>         Attachments: HIVE-11229.1.patch
> [~ekoifman] raised a theoretical issue with the streaming mutation API (HIVE-10165) where
worker nodes operating in a distributed cluster might overwhelm a meta store while trying
to obtain partition locks. Although this does not happen in practice (see HIVE-11228), the
API does communicate with the meta store in this manner to obtain partition paths and create
new partitions. Therefore the issue described does in fact exist in the current implementation,
albeit in a different code path. I’d like to make such communication optional like so:
> * When the user chooses not to create partitions on demand, no meta store connection
will be created in the {{MutationCoordinators}}. Additionally, partition paths will be resolved
using {{org.apache.hadoop.hive.metastore.Warehouse.getPartitionPath(Path, LinkedHashMap<String,
String>)}} which should be suitable so long as standard Hive partition layouts are followed.
> * If the user does choose to create partitions on demand then the system will operate
as is does currently; using the meta store to both issue {{add_partition}} events and look
up partition meta data.
> * The documentation will be updated to describe these behaviours and outline alternative
approaches to collecting affected partition names and creating partitions in a less intensive
> Side note for follow up: The parameter names {{tblName}} and {{dbName}} seem to be the
wrong way around on the method {{org.apache.hadoop.hive.metastore.IMetaStoreClient.getPartition(String,
String, List<String>)}}.

This message was sent by Atlassian JIRA

View raw message