hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Dere (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join
Date Tue, 30 Oct 2018 17:35:02 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669100#comment-16669100
] 

Jason Dere commented on HIVE-20839:
-----------------------------------

Issue seems to be that there are several column expressions that end up being converted to
constant value 'SSS' during constant folding. Some of these constant columns end up being
pruned from the query, and others might be join keys which are required in the query. The
problem was that ExprNodeDescUtils.resolveJoinKeysAsRSColumns() (which is used to setup the
join key expressions in the case of dynamically partitioned hashjoin) was trying to resolve
the join key expression for the 'SSS' column and picked the wrong column - it was also constant
'SSS', but it was actually a different column that ended up getting pruned.

The ExprNodeConstDesc actually has a getFoldedFromCol() field that shows what column this
constant expression has been folded from, I think this can be used to help resolve to the
correct expression.

> "Cannot find field" error during dynamically partitioned hash join
> ------------------------------------------------------------------
>
>                 Key: HIVE-20839
>                 URL: https://issues.apache.org/jira/browse/HIVE-20839
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: Jason Dere
>            Assignee: Jason Dere
>            Priority: Major
>
> Occurs in some cases in the non-CBO optimized queries, either if CBO is disabled or has
failed due to error.
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2 (1539092085144_8944_1085_28_000996_2)]
tez.ReduceRecordProcessor: Hit error while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2 (1539092085144_8944_1085_28_000996_2)]
tez.TezProcessor: java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9, 10:_col10, 11:_col11,
12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17, 17:_col18, 18:_col19, 19:_col20, 20:_col21,
21:_col22, 22:_col23, 23:_col24, 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30,
30:_col31, 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38, 38:_col39,
39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45, 45:_col46, 46:_col47, 47:_col48,
48:_col49, 49:_col50, 50:_col51, 51:_col52, 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57,
57:_col58, 58:_col59, 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75, 73:_col76, 74:_col77,
75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82, 80:_col83, 81:_col84, 82:_col85, 83:_col86,
84:_col87, 85:_col88, 86:_col89, 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95,
93:_col96, 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102, 100:_col103,
101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108, 106:_col109, 107:_col110,
108:_col111, 109:_col112, 110:_col113, 111:_col114, 112:_col115, 113:_col116, 114:_col117,
115:_col118, 116:_col119, 117:_col120, 118:_col121, 119:_col122, 120:_col123, 121:_col124,
122:_col125, 123:_col126, 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131,
129:_col132, 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144, 142:_col145,
143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150, 148:_col151, 149:_col152,
150:_col153, 151:_col154, 152:_col155, 153:_col156, 154:_col157, 155:_col158, 156:_col159,
157:_col160, 158:_col161, 159:_col162, 160:_col163, 161:_col164, 162:_col165, 163:_col166,
164:_col167, 165:_col168, 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
> at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message