tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jianfeng (Jeff) Zhang" <jzh...@hortonworks.com>
Subject Re: hive 0.14.1 + tez 0.4.0 sometimes throws NPE on small data
Date Mon, 12 Jan 2015 08:04:28 GMT
Hi Alexander,

Hive mail list would be a more proper place for this kind of question, it
looks like an issue from Hive's Processor.



Best Regards,
Jeff Zhang


On Sat, Jan 10, 2015 at 3:59 AM, Alexander Pivovarov <apivovarov@gmail.com>
wrote:

> I noticed that if amount of data which should be sent to reducers is small
> then tez might throw NPE.
>
> workaround is to set number of reducers to some small value.
> e.g. if initially I have
> set hive.exec.reducers.max=25;
> then to solve the issue I might try
> set hive.exec.reducers.max=13;
> or even smaller value
>
> I'm using hdp-2.1.5 (which is hive-0.14.1 + tez-0.4.0)
>
> Map 1: 17/21    Map 11: 1/1     Map 14: 4/4     Map 16: 4/4     Map 18:
> 4/4     Map 19: 1/1     Map 21: 1/1     Map 22: 5/5     Map 23: 18/21   Map
> 24: 1/1     Map 4: 21/21    Map 6: 1/1      Map 9: 1/1    1
> Status: Failed
> Vertex re-running, vertexName=Map 6, vertexId=vertex_1420058390346_
> 15040_1_20
> Vertex re-running, vertexName=Map 18,
> vertexId=vertex_1420058390346_15040_1_10
> Vertex re-running, vertexName=Map 22,
> vertexId=vertex_1420058390346_15040_1_14
> Vertex re-running, vertexName=Map 4,
> vertexId=vertex_1420058390346_15040_1_22
> Vertex re-running, vertexName=Map 9,
> vertexId=vertex_1420058390346_15040_1_17
> Vertex re-running, vertexName=Map 19,
> vertexId=vertex_1420058390346_15040_1_02
> Vertex re-running, vertexName=Map 24,
> vertexId=vertex_1420058390346_15040_1_09
> Vertex re-running, vertexName=Map 1,
> vertexId=vertex_1420058390346_15040_1_24
> Vertex re-running, vertexName=Map 14,
> vertexId=vertex_1420058390346_15040_1_15
> Vertex re-running, vertexName=Map 16,
> vertexId=vertex_1420058390346_15040_1_12
> Vertex re-running, vertexName=Map 14,
> vertexId=vertex_1420058390346_15040_1_15
> Vertex re-running, vertexName=Map 6,
> vertexId=vertex_1420058390346_15040_1_20
> Vertex re-running, vertexName=Map 18,
> vertexId=vertex_1420058390346_15040_1_10
> Vertex re-running, vertexName=Map 22,
> vertexId=vertex_1420058390346_15040_1_14
> Vertex re-running, vertexName=Map 21,
> vertexId=vertex_1420058390346_15040_1_19
> Vertex re-running, vertexName=Map 16,
> vertexId=vertex_1420058390346_15040_1_12
> Vertex re-running, vertexName=Map 24,
> vertexId=vertex_1420058390346_15040_1_09
> Vertex re-running, vertexName=Map 11,
> vertexId=vertex_1420058390346_15040_1_06
> Vertex failed, vertexName=Reducer 17,
> vertexId=vertex_1420058390346_15040_1_11, diagnostics=[Task failed,
> taskId=task_1420058390346_15040_1_11_000007,
> diagnostics=[AttemptID:attempt_1420058390346_15040_1_11}
>         at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:188)
>         at
>
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
>         at
> org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:564)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
>         at
> org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:553)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{"reducesinkkey0":61419,"reducesinkkey1":"2011-10-02 13:04:53"}
>         at
>
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:349)
>         at
>
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:228)
>         at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
>         ... 6 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{"reducesinkkey0":61419,"reducesinkkey1":"2011-10-02
> 13:04:53","reducesinkkey2":"2011-11-0}
>         at
>
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:334)
>         ... 8 more
> Caused by: java.lang.NullPointerException
>         at
>
> org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:341)
>         at
> org.apache.hadoop.hive.ql.exec.PTFOperator.processOp(PTFOperator.java:115)
>         at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
>         at
>
> org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
>         at
>
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:325)
>         ... 8 more
>
> Container released by application,
> AttemptID:attempt_1420058390346_15040_1_11_000007_1 Info:Node blacklisted,
> AttemptID:attempt_1420058390346_15040_1_11_000007_2 Info:Node blacklisted
> Container released by application,
> AttemptID:attempt_1420058390346_15040_1_11_000007_3 Info:Error:
> java.lang.RuntimeException: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: H}
>         at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:188)
>         at
>
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
>         at
> org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:564)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
>         at
> org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:553)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{"reducesinkkey0":61419,"reducesinkkey1":"2011-10-02 13:04:53"}
>         at
>
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:349)
>         at
>
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:228)
>         at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
>         ... 6 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{"reducesinkkey0":61419,"reducesinkkey1":"2011-10-02
> 13:04:53","reducesinkkey2":"2011-11-0}
>         at
>
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:334)
>         ... 8 more
> Caused by: java.lang.NullPointerException
>         at
>
> org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:341)
>         at
> org.apache.hadoop.hive.ql.exec.PTFOperator.processOp(PTFOperator.java:115)
>         at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
>         at
>
> org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
>         at
>
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:325)
>         ... 8 more
>
> Container released by application,
> AttemptID:attempt_1420058390346_15040_1_11_000007_4 Info:Error:
> java.lang.RuntimeException: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: H}
>         at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:188)
>         at
>
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
>         at
> org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:564)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
>         at
> org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:553)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{"reducesinkkey0":61419,"reducesinkkey1":"2011-10-02 13:04:53"}
>         at
>
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:349)
>         at
>
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:228)
>         at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
>         ... 6 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{"reducesinkkey0":61419,"reducesinkkey1":"2011-10-02
> 13:04:53","reducesinkkey2":"2011-11-0}
>         at
>
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:334)
>         ... 8 more
> Caused by: java.lang.NullPointerException
>         at
>
> org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:341)
>         at
> org.apache.hadoop.hive.ql.exec.PTFOperator.processOp(PTFOperator.java:115)
>         at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
>         at
>
> org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
>         at
>
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:325)
>         ... 8 more
>
> Container released by application,
> AttemptID:attempt_1420058390346_15040_1_11_000007_5 Info:Error:
> java.lang.RuntimeException: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: H}
>         at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:188)
>         at
>
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
>         at
> org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:564)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
>         at
> org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:553)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{"reducesinkkey0":61419,"reducesinkkey1":"2011-10-02 13:04:53"}
>         at
>
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:349)
>         at
>
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:228)
>         at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
>         ... 6 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{"reducesinkkey0":61419,"reducesinkkey1":"2011-10-02
> 13:04:53","reducesinkkey2":"2011-11-0}
>         at
>
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:334)
>         ... 8 more
> Caused by: java.lang.NullPointerException
>         at
>
> org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:341)
>         at
> org.apache.hadoop.hive.ql.exec.PTFOperator.processOp(PTFOperator.java:115)
>         at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
>         at
>
> org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
>         at
>
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:325)
>         ... 8 more
> ], Vertex failed as one or more tasks failed. failedTasks:1]
> Vertex killed, vertexName=Reducer 13,
> vertexId=vertex_1420058390346_15040_1_03, diagnostics=[Vertex received Kill
> while in RUNNING state., Vertex killed as other vertex failed.
> failedTasks:0]
> Vertex killed, vertexName=Reducer 3,
> vertexId=vertex_1420058390346_15040_1_00, diagnostics=[Vertex received Kill
> while in RUNNING state., Vertex killed as other vertex failed.
> failedTasks:0]
> Vertex killed, vertexName=Reducer 25,
> vertexId=vertex_1420058390346_15040_1_08, diagnostics=[Vertex received Kill
> while in RUNNING state., Vertex killed as other vertex failed.
> failedTasks:0]
> Vertex killed, vertexName=Reducer 15,
> vertexId=vertex_1420058390346_15040_1_13, diagnostics=[Vertex received Kill
> while in RUNNING state., Vertex killed as other vertex failed.
> failedTasks:0]
> Vertex killed, vertexName=Reducer 20,
> vertexId=vertex_1420058390346_15040_1_01, diagnostics=[Vertex received Kill
> while in RUNNING state., Vertex killed as other vertex failed.
> failedTasks:0]
> Vertex killed, vertexName=Reducer 8,
> vertexId=vertex_1420058390346_15040_1_07, diagnostics=[Vertex received Kill
> while in RUNNING state., Vertex killed as other vertex failed.
> failedTasks:0]
> Vertex killed, vertexName=Reducer 12,
> vertexId=vertex_1420058390346_15040_1_04, diagnostics=[Vertex received Kill
> while in RUNNING state., Vertex killed as other vertex failed.
> failedTasks:0]
> Vertex killed, vertexName=Map 23, vertexId=vertex_1420058390346_15040_1_05,
> diagnostics=[Vertex received Kill while in RUNNING state., Vertex killed as
> other vertex failed. failedTasks:0]
> Vertex killed, vertexName=Map 1, vertexId=vertex_1420058390346_15040_1_24,
> diagnostics=[Vertex received Kill while in RUNNING state., Vertex killed as
> other vertex failed. failedTasks:0]
> Vertex killed, vertexName=Reducer 5,
> vertexId=vertex_1420058390346_15040_1_21, diagnostics=[Vertex received Kill
> while in RUNNING state., Vertex killed as other vertex failed.
> failedTasks:0]
> Vertex killed, vertexName=Reducer 2,
> vertexId=vertex_1420058390346_15040_1_23, diagnostics=[Vertex received Kill
> while in RUNNING state., Vertex killed as other vertex failed.
> failedTasks:0]
> Vertex killed, vertexName=Reducer 10,
> vertexId=vertex_1420058390346_15040_1_16, diagnostics=[Vertex received Kill
> while in RUNNING state., Vertex killed as other vertex failed.
> failedTasks:0]
> Vertex killed, vertexName=Reducer 7,
> vertexId=vertex_1420058390346_15040_1_18, diagnostics=[Vertex received Kill
> while in RUNNING state., Vertex killed as other vertex failed.
> failedTasks:0]
> DAG failed due to vertex failure. failedVertices:1 killedVertices:13
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message