hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "anishek (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20545) Ability to exclude potentially large parameters in HMS Notifications
Date Wed, 10 Oct 2018 06:31:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644514#comment-16644514
] 

anishek commented on HIVE-20545:
--------------------------------

Thanks [~vihangk1] for the details. I wanted to understand some more on the thirft object
before/after serialization, Since  DBNotificationListener is on the metastore, there should
not be additional serialization of thrift objets there, however as the json grows larger definitely
there is additional overhead of space on rdbms + in memory to keep them, which i think would
be good to reduce. 

So the serialization i assume you are mentioning is the json serialization of the java objects?
There is another effort we are looking at via HIVE-20679 to see if we can have zipped message
in db to reduce the network transfer and rdbms overhead of these messages. 

Just out of curiosity these stats i assume will help impala to better plan the query ? if
yes how will the same work on target warehouse after replication ?

> Ability to exclude potentially large parameters in HMS Notifications
> --------------------------------------------------------------------
>
>                 Key: HIVE-20545
>                 URL: https://issues.apache.org/jira/browse/HIVE-20545
>             Project: Hive
>          Issue Type: Improvement
>          Components: Metastore
>    Affects Versions: 3.1.0, 4.0.0
>            Reporter: Bharathkrishna Guruvayoor Murali
>            Assignee: Bharathkrishna Guruvayoor Murali
>            Priority: Major
>         Attachments: HIVE-20545.1.patch, HIVE-20545.2.patch, HIVE-20545.3.branch-3.patch,
HIVE-20545.3.patch, HIVE-20545.4.patch, HIVE-20545.6.patch, HIVE-20545.7.patch
>
>
> Clients can add large-sized parameters in Table/Partition objects. So we need to enable adding
regex patterns through HiveConf to match parameters to be filtered from table and partition
objects before serialization in HMS notifications.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message