hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11355) Hive on tez: memory manager for sort buffers (input/output) and operators
Date Fri, 07 Aug 2015 19:37:45 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662319#comment-14662319
] 

Sergey Shelukhin commented on HIVE-11355:
-----------------------------------------

in GBO - why isn't it checking for flushes anymore if memoryNeeded is specified?
createEdgeProperty nit - memoryManagerEnabled is only used in one case block but it is retrieved
for all of them.
dummyStoreOp.setConf(new DummyStoreDesc()); - what is this for?
{noformat}
Invalid configuration could cause OutOfMemory issues at runtime. Configuration "
            + HiveConf.ConfVars.HIVECONVERTJOINNOCONDITIONALTASKTHRESHOLD + " = "
            + mapJoinsTotalAvailableMemory + " is greater than configured container size:
"
            + totalAvailableMemory + ". Check your tez/yarn settings");
{noformat}
Will this happen to many people? When the threshold is set to some arbitrary high value to
force map join, this may happen, even though this value will not actually be the table size
in reality. IIRC I saw it in some tests.
To be continued


> Hive on tez: memory manager for sort buffers (input/output) and operators
> -------------------------------------------------------------------------
>
>                 Key: HIVE-11355
>                 URL: https://issues.apache.org/jira/browse/HIVE-11355
>             Project: Hive
>          Issue Type: Improvement
>          Components: Tez
>    Affects Versions: 2.0.0
>            Reporter: Vikram Dixit K
>            Assignee: Vikram Dixit K
>         Attachments: HIVE-11355.1.patch, HIVE-11355.2.patch, HIVE-11355.3.patch
>
>
> We need to better manage the sort buffer allocations to ensure better performance. Also,
we need to provide configurations to certain operators to stay within memory limits.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message