hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zoltan Haindrich (JIRA)" <>
Subject [jira] [Commented] (HIVE-19824) Improve online datasize estimations for MapJoins
Date Mon, 11 Jun 2018 14:48:00 GMT


Zoltan Haindrich commented on HIVE-19824:

I've not seen the vectorized one in action...but I've just looked into it: they also seem
to have similar overhead; I think for those datastructures the overhead is a little but larger;
since it stores 2 refs + a hashcode for every lookup position - so its around: {{dataSize
+ 24 * log2UP(rowNum / loadFactor)}}

> Improve online datasize estimations for MapJoins
> ------------------------------------------------
>                 Key: HIVE-19824
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Zoltan Haindrich
>            Assignee: Zoltan Haindrich
>            Priority: Major
>         Attachments: HIVE-19824.01wip01.patch, HIVE-19824.01wip01.patch
> Statistics.datasize() only accounts for "real" data size; but for example handling 1M
rows might introduce some datastructure overhead...if the "real" data is small - even this
overhead might become the real memory usage
> for 6.5M rows of (int,int) the estimation is 52MB
> in reality this eats up ~260MB from which 210MB is used to service the hashmap functionality
to that many rows.

This message was sent by Atlassian JIRA

View raw message