nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Yu <tsunbo...@gmail.com>
Subject Get IOException occasionally
Date Tue, 14 May 2019 23:37:16 GMT
Hi all,
I need help finding the root cause on this issue.
I occasionally get the following exception from SelectHiveQL and PutHiveQL
processor.
Sometimes the queries executed without any issue but next minute this
exception could happen.
Once it happened, by restarting only the NiFi does not resolve the issue.
I have to reboot the Linux VM of the node and remove the flow.xml.gz file
then restart nifi node.

I am using NIFI version 1.7.1 in cluster mode with total 8 nodes

Here is the stacktrace
2019-05-14 20:37:30,937 ERROR [Timer-Driven Process Thread-6]
o.a.nifi.processors.hive.SelectHiveQL
SelectHiveQL[id=365c2597-9483-302c-90d0-c7d2d7ce471d] Unable to execute
HiveQL select query select max(version) as max_partition, min(version) as
min_partition from Table_A for
StandardFlowFileRecord[uuid=9253b1ab-b4a2-4547-b695-4dbf5f964ca2,claim=,offset=0,name=done,size=0]
due to org.apache.nifi.processor.exception.ProcessException: Error during
database query or conversion of records.; routing to failure:
org.apache.nifi.processor.exception.ProcessException: Error during database
query or conversion of records.
org.apache.nifi.processor.exception.ProcessException: Error during database
query or conversion of records.
at
org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$2(SelectHiveQL.java:407)
at
org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$627/1348460623.process(Unknown
Source)
at
org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2625)
at
org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:395)
at
org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$0(SelectHiveQL.java:285)
at
org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$622/285174376.execute(Unknown
Source)
at
org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
at
org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106)
at
org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:285)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: java.io.IOException:
java.lang.RuntimeException: native snappy library not available
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:330)
at
org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207)
at
org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207)
at
org.apache.nifi.util.hive.HiveJdbcCommon.convertToCsvStream(HiveJdbcCommon.java:355)
at
org.apache.nifi.util.hive.HiveJdbcCommon.convertToCsvStream(HiveJdbcCommon.java:322)
at
org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$2(SelectHiveQL.java:401)
... 18 common frames omitted
2019-05-14 20:37:30,938 ERROR [Timer-Driven Process Thread-6]
o.a.nifi.processors.hive.SelectHiveQL
SelectHiveQL[id=365c2597-9483-302c-90d0-c7d2d7ce471d]
org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$622/285174376@54eb300d
failed to process due to
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=e2688819-edc9-406a-b5da-050ac072cd43,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1557687270864-64540,
container=default, section=28], offset=81777,
length=280],offset=0,name=done,size=280] transfer relationship not
specified; rolling back session:
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=e2688819-edc9-406a-b5da-050ac072cd43,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1557687270864-64540,
container=default, section=28], offset=81777,
length=280],offset=0,name=done,size=280] transfer relationship not specified
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=e2688819-edc9-406a-b5da-050ac072cd43,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1557687270864-64540,
container=default, section=28], offset=81777,
length=280],offset=0,name=done,size=280] transfer relationship not specified
at
org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:264)
at
org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:335)
at
org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:115)
at
org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106)
at
org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:285)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)


Thanks
David

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message