hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "kumar ravi (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-23412) Filter applied in the "LEFT JOIN" condition is pulling a null batch on last reducer (map join vectorizer operator) and throwing exception as "Hive Runtime Error while processing vector batch" at the map join during the query execution.
Date Fri, 08 May 2020 15:54:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-23412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102688#comment-17102688
] 

kumar ravi commented on HIVE-23412:
-----------------------------------

select count(t.abc),
 DATE(t.aaa) as ashdjahsdj,
 t.bbb,
 t.ccc,
 sum(b.eee / 100) as amount,
 (case
 when b.eee / 100 >= 0 and b.eee / 100 <= 1000 then '0-1000'
 when b.eee / 100 > 1000 and b.eee / 100 <= 2000 then '1001-2000'
 when b.eee / 100 > 2000 then '2000 above' end) as slab,
 b.eee,
 b.p_eee,
 g.eee,
 g.lol,
 h.how,
 h.pg_acquirer
from www.ashdjahsdj t
 JOIN www.shjhs b ON b.abc = t.abc
 JOIN www.sdjsd g ON g.abc = t.abc
 LEFT JOIN www.sdsd h ON t.abc = h.pp_txn_id and h.year = '2020'
where t.state = "COMPLETED"
 and b.eee in ('asdasd', 'feffe', 'vfvfv', 'fdvfv','xdfdf')
 and (t.ccc == 0
 or ((t.ccc & shiftleft(1, 0) == 0)
 and (t.ccc & shiftleft(1, 1) == 0))
 )
 and t.aaa >= '2020-04-01'
 and t.aaa < '2020-05-01'
 and t.year = '2020' and t.month = '04'
 and b.year = '2020' and b.month = '04'
 and g.year = '2020' and g.month = '04'
group by g.lol, b.eee, b.p_eee, g.eee, t.bbb, t.ccc, h.how, h.pg_acquirer,
 DATE(t.aaa), case
 when b.eee / 100 >= 0 and b.eee / 100 <= 1000 then '0-1000'
 when b.eee / 100 > 1000 and b.eee / 100 <= 2000 then '1001-2000'
 when b.eee / 100 > 2000 then '2000 above' end;

> Filter applied in the "LEFT JOIN" condition is pulling a null batch on last reducer (map
join vectorizer operator) and throwing exception as "Hive Runtime Error while processing vector
batch" at the map join during the query execution.
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-23412
>                 URL: https://issues.apache.org/jira/browse/HIVE-23412
>             Project: Hive
>          Issue Type: Bug
>          Components: Vectorization
>    Affects Versions: 3.1.0
>            Reporter: kumar ravi
>            Priority: Major
>         Attachments: Query.rtf
>
>
> Filter applied in the "LEFT JOIN" condition is pulling a null batch on last reducer (map
join vectorizer operator) and throwing exception as "Hive Runtime Error while processing vector
batch" at the map join during the query execution.
> {{ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask.
Vertex re-running, vertexName=Map 2, vertexId=vertex_1588769098286_8402_1_00Vertex re-running,
vertexName=Map 1, vertexId=vertex_1588769098286_8402_1_02Vertex re-running, vertexName=Map
4, vertexId=vertex_1588769098286_8402_1_04Vertex failed, vertexName=Reducer 6, vertexId=vertex_1588769098286_8402_1_05,
diagnostics=[Task failed, taskId=task_1588769098286_8402_1_05_000604, diagnostics=[TaskAttempt
0 failed, info=[Error: Error while running task ( failure ) : attempt_1588769098286_8402_1_05_000604_0:java.lang.RuntimeException:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing vector batch (tag=3) (vectorizedVertexNum 5)}}{{ at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)}}{{
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)}}{{ at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)}}{{
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)}}{{
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)}}{{
at java.security.AccessController.doPrivileged(Native Method)}}{{ at javax.security.auth.Subject.doAs(Subject.java:422)}}{{
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)}}{{
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)}}{{
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)}}{{
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)}}{{ at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)}}{{
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)}}{{
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)}}{{
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}{{ at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}{{ at java.lang.Thread.run(Thread.java:748)}}{{Caused
by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing vector batch (tag=3) (vectorizedVertexNum 5)}}{{ at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:401)}}{{
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:249)}}{{
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:318)}}{{
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)}}{{
... 16 more}}{{Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing vector batch (tag=3) (vectorizedVertexNum 5)}}{{ at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:503)}}{{
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:392)}}{{
... 19 more}}{{Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception
from VectorMapJoinOperator : null}}{{ at org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:539)}}{{
at org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.process(VectorMapJoinOperator.java:235)}}{{
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:490)}}{{
... 20 more}}{{Caused by: java.lang.NullPointerException}}{{ at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getFilterTag(CommonJoinOperator.java:802)}}{{
at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:600)}}{{
at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genJoinObject(CommonJoinOperator.java:533)}}{{
at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:936)}}{{
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:526)}}{{ ...
22 more}}{{], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) : attempt_1588769098286_8402_1_05_000604_1:java.lang.RuntimeException:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing vector batch (tag=3) (vectorizedVertexNum 5)}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message