hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aihua Xu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20153) Count and Sum UDF consume more memory in Hive 2+
Date Sun, 15 Jul 2018 20:42:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16544665#comment-16544665
] 

Aihua Xu commented on HIVE-20153:
---------------------------------

[~gopalv] If you want to take a look, feel free to take it. Otherwise, I will be on PTO for
a week and will investigate after that.

> Count and Sum UDF consume more memory in Hive 2+
> ------------------------------------------------
>
>                 Key: HIVE-20153
>                 URL: https://issues.apache.org/jira/browse/HIVE-20153
>             Project: Hive
>          Issue Type: Bug
>          Components: UDF
>    Affects Versions: 2.3.2
>            Reporter: Szehon Ho
>            Assignee: Aihua Xu
>            Priority: Major
>         Attachments: Screen Shot 2018-07-12 at 6.41.28 PM.png
>
>
> While playing with Hive2, we noticed that queries with a lot of count() and sum() aggregations
run out of memory on Hadoop side where they worked before in Hive1. 
> In many queries, we have to double the Mapper Memory settings (in our particular case
mapreduce.map.java.opts from -Xmx2000M to -Xmx4000M), it makes it not so easy to upgrade
to Hive 2.
> Taking heap dump, we see one of the main culprit is the field 'uniqueObjects' in GeneraicUDAFSum
and GenericUDAFCount, which was added to support Window functions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message