hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
Date Thu, 16 Aug 2018 18:35:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582912#comment-16582912
] 

Hive QA commented on HIVE-20321:
--------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 57s{color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 11s{color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 39s{color}
| {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 18s{color} | {color:blue}
ql in master has 2305 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  1s{color} |
{color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 30s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  9s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  9s{color} | {color:green}
the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 41s{color} | {color:red}
ql: The patch generated 50 new + 144 unchanged - 15 fixed = 194 total (was 159) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 26s{color} | {color:red}
ql generated 4 new + 2304 unchanged - 1 fixed = 2308 total (was 2305) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  1s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 13s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 42s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperBase.equals(Object)
is unusual  At VectorHashKeyWrapperBase.java:VectorHashKeyWrapperBase.java:[line 77] |
|  |  org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperBatch.getVectorHashKeyWrappers()
may expose internal representation by returning VectorHashKeyWrapperBatch.vectorHashKeyWrappers
 At VectorHashKeyWrapperBatch.java:by returning VectorHashKeyWrapperBatch.vectorHashKeyWrappers
 At VectorHashKeyWrapperBatch.java:[line 89] |
|  |  org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperSingleLong defines
equals but not hashCode  At VectorHashKeyWrapperSingleLong.java:hashCode  At VectorHashKeyWrapperSingleLong.java:[lines
43-49] |
|  |  org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperTwoLong defines equals
but not hashCode  At VectorHashKeyWrapperTwoLong.java:hashCode  At VectorHashKeyWrapperTwoLong.java:[lines
55-63] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03)
x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-13273/dev-support/hive-personality.sh
|
| git revision | master / ddc2465 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-13273/yetus/diff-checkstyle-ql.txt
|
| findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-13273/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-13273/yetus.txt |
| Powered by | Apache Yetus    http://yetus.apache.org |


This message was automatically generated.



> Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
> ---------------------------------------------------------------------------------
>
>                 Key: HIVE-20321
>                 URL: https://issues.apache.org/jira/browse/HIVE-20321
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Gopal V
>            Assignee: Matt McCline
>            Priority: Major
>         Attachments: HIVE-20321.01.patch, HIVE-20321.02.patch, HIVE-20321.03.patch, HIVE-20321.04.patch
>
>
> With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper is bigger
than the low Xmx JVMs.
> {code}
> ***** 64-bit VM: **********************************************************
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals:
>  OFFSET  SIZE                                                                     TYPE
DESCRIPTION                                  VALUE
>       0    16                                                                       
  (object header)                              N/A
>      16     4                                                                      int
VectorHashKeyWrapper.hashcode                N/A
>      20     4                                                                       
  (alignment/padding gap)                     
>      24     8                                                                   long[]
VectorHashKeyWrapper.longValues              N/A
>      32     8                                                                 double[]
VectorHashKeyWrapper.doubleValues            N/A
>      40     8                                                                 byte[][]
VectorHashKeyWrapper.byteValues              N/A
>      48     8                                                                    int[]
VectorHashKeyWrapper.byteStarts              N/A
>      56     8                                                                    int[]
VectorHashKeyWrapper.byteLengths             N/A
>      64     8                   org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[]
VectorHashKeyWrapper.decimalValues           N/A
>      72     8                                                     java.sql.Timestamp[]
VectorHashKeyWrapper.timestampValues         N/A
>      80     8                 org.apache.hadoop.hive.common.type.HiveIntervalDayTime[]
VectorHashKeyWrapper.intervalDayTimeValues   N/A
>      88     8                                                                boolean[]
VectorHashKeyWrapper.isNull                  N/A
>      96     8   org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext
VectorHashKeyWrapper.hashCtx                 N/A
> Instance size: 104 bytes
> Space losses: 4 bytes internal + 0 bytes external = 4 bytes total
> {code}
> Pulling this up to a parent class allows for this to be cut down to 32 bytes for the
single column case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message