hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt McCline (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-12208) Vectorized JOIN NPE on dynamically partitioned hash-join + map-join
Date Tue, 03 Nov 2015 00:08:27 GMT

    [ https://issues.apache.org/jira/browse/HIVE-12208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14986328#comment-14986328
] 

Matt McCline commented on HIVE-12208:
-------------------------------------

Attached a patch that is a bug detector.  It adds default cases for the switch stmts that
create the native vector map join classes.  And assert the hash table is not null.

If this patch doesn't trigger, then the only other idea I have is I did have a problem with
MapJoinBytesTableContainer.isSupportedKey rejecting using the MapJoinBytesTableContainer when
I had already approved it for native vector map join.  But that should have resulted in a
cast error.

> Vectorized JOIN NPE on dynamically partitioned hash-join + map-join
> -------------------------------------------------------------------
>
>                 Key: HIVE-12208
>                 URL: https://issues.apache.org/jira/browse/HIVE-12208
>             Project: Hive
>          Issue Type: Bug
>          Components: Vectorization
>    Affects Versions: 2.0.0
>            Reporter: Gopal V
>            Assignee: Matt McCline
>         Attachments: HIVE-12208.01.patch, query82.txt
>
>
> TPC-DS Q82 with reducer vectorized join optimizations
> {code}
>   Reducer 5 <- Map 1 (CUSTOM_SIMPLE_EDGE), Map 2 (CUSTOM_SIMPLE_EDGE), Map 3 (BROADCAST_EDGE),
Map 4 (CUSTOM_SIMPLE_EDGE)
> {code}
> {code}
> set hive.optimize.dynamic.partition.hashjoin=true;
> set hive.vectorized.execution.reduce.enabled=true;
> set hive.mapjoin.hybridgrace.hashtable=false;
> select  i_item_id
>        ,i_item_desc
>        ,i_current_price
>  from item, inventory, date_dim, store_sales
>  where i_current_price between 30 and 30+30
>  and inv_item_sk = i_item_sk
>  and d_date_sk=inv_date_sk
>  and d_date between '2002-05-30' and '2002-07-30'
>  and i_manufact_id in (437,129,727,663)
>  and inv_quantity_on_hand between 100 and 500
>  and ss_item_sk = i_item_sk
>  group by i_item_id,i_item_desc,i_current_price
>  order by i_item_id
>  limit 100
> {code}
> possibly a trivial plan setup issue, since the NPE is pretty much immediate.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
> 	at org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:368)
> 	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:852)
> 	at org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardBigTableBatch(VectorMapJoinGenerateResultOperator.java:603)
> 	at org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:362)
> 	... 19 more
> Caused by: java.lang.NullPointerException
> 	at org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.commonSetup(VectorMapJoinInnerGenerateResultOperator.java:112)
> 	at org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:96)
> 	... 22 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message