hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Vary (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-17990) Add Thrift and DB storage for Schema Registry objects
Date Fri, 08 Dec 2017 13:36:01 GMT

    [ https://issues.apache.org/jira/browse/HIVE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283516#comment-16283516
] 

Peter Vary commented on HIVE-17990:
-----------------------------------

Hi [~alangates],

Thanks for working on this. I have taken a quick look at this patch, and have some questions:
AFAIK the current MetaStore API is quite verbose - and because of the limitations of Thrift
- every object is serialized in the response as many times as it is referenced in the result.
This causes serious memory issues in HS2 and/or HMS when running queries on tens of thousands
of partitions.
If I understand the proposal correctly, this specific change adds even more data to the response,
and in most of the cases this data will be the same for every partition in the request. This
is true for the initial changes and even more so for the eventual results.

I am totally positive for the Schema Registry integration, and if we could find a way to not
to increase the memory pressure, or we have plans for help this in the future, I would be
happy.

Also it would be nice to have unit tests for the new HMS API interface methods in this jira,
or one follow-up jira.

Please feel free to correct me if I am wrong in anything stated above.
Thanks,
Peter

> Add Thrift and DB storage for Schema Registry objects
> -----------------------------------------------------
>
>                 Key: HIVE-17990
>                 URL: https://issues.apache.org/jira/browse/HIVE-17990
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Standalone Metastore
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>         Attachments: Adding-Schema-Registry-to-Metastore.pdf
>
>
> This JIRA tracks changes to Thrift, RawStore, and DB scripts to support objects in the
Schema Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message