nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Weeks <swe...@weeksconsulting.us>
Subject Re: Get IOException occasionally
Date Wed, 15 May 2019 12:54:03 GMT
There are several complications using Snappy but from what I can tell the SelectHiveQL Processor
isn't trying to use it and instead the Hive JDBC Driver is trying to use it. Are you running
a specific vendors distribution of NiFi? The reason I'm asking is when I go look at the source
for the Hive JDBC Driver Class org.apache.hive.jdbc.Utils.verifySuccess where the error is
being raised none of the line numbers are matching up with Apache. It may be that the error
is actually coming from Hive not NiFi.

Thanks
Shawn

´╗┐On 5/14/19, 6:37 PM, "David Yu" <tsunbonyu@gmail.com> wrote:

    Hi all,
    I need help finding the root cause on this issue.
    I occasionally get the following exception from SelectHiveQL and PutHiveQL
    processor.
    Sometimes the queries executed without any issue but next minute this
    exception could happen.
    Once it happened, by restarting only the NiFi does not resolve the issue.
    I have to reboot the Linux VM of the node and remove the flow.xml.gz file
    then restart nifi node.
    
    I am using NIFI version 1.7.1 in cluster mode with total 8 nodes
    
    Here is the stacktrace
    2019-05-14 20:37:30,937 ERROR [Timer-Driven Process Thread-6]
    o.a.nifi.processors.hive.SelectHiveQL
    SelectHiveQL[id=365c2597-9483-302c-90d0-c7d2d7ce471d] Unable to execute
    HiveQL select query select max(version) as max_partition, min(version) as
    min_partition from Table_A for
    StandardFlowFileRecord[uuid=9253b1ab-b4a2-4547-b695-4dbf5f964ca2,claim=,offset=0,name=done,size=0]
    due to org.apache.nifi.processor.exception.ProcessException: Error during
    database query or conversion of records.; routing to failure:
    org.apache.nifi.processor.exception.ProcessException: Error during database
    query or conversion of records.
    org.apache.nifi.processor.exception.ProcessException: Error during database
    query or conversion of records.
    at
    org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$2(SelectHiveQL.java:407)
    at
    org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$627/1348460623.process(Unknown
    Source)
    at
    org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2625)
    at
    org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:395)
    at
    org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$0(SelectHiveQL.java:285)
    at
    org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$622/285174376.execute(Unknown
    Source)
    at
    org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
    at
    org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106)
    at
    org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:285)
    at
    org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
    at
    org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
    at
    org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    Caused by: java.sql.SQLException: java.io.IOException:
    java.lang.RuntimeException: native snappy library not available
    at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
    at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
    at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:330)
    at
    org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207)
    at
    org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207)
    at
    org.apache.nifi.util.hive.HiveJdbcCommon.convertToCsvStream(HiveJdbcCommon.java:355)
    at
    org.apache.nifi.util.hive.HiveJdbcCommon.convertToCsvStream(HiveJdbcCommon.java:322)
    at
    org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$2(SelectHiveQL.java:401)
    ... 18 common frames omitted
    2019-05-14 20:37:30,938 ERROR [Timer-Driven Process Thread-6]
    o.a.nifi.processors.hive.SelectHiveQL
    SelectHiveQL[id=365c2597-9483-302c-90d0-c7d2d7ce471d]
    org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$622/285174376@54eb300d
    failed to process due to
    org.apache.nifi.processor.exception.FlowFileHandlingException:
    StandardFlowFileRecord[uuid=e2688819-edc9-406a-b5da-050ac072cd43,claim=StandardContentClaim
    [resourceClaim=StandardResourceClaim[id=1557687270864-64540,
    container=default, section=28], offset=81777,
    length=280],offset=0,name=done,size=280] transfer relationship not
    specified; rolling back session:
    org.apache.nifi.processor.exception.FlowFileHandlingException:
    StandardFlowFileRecord[uuid=e2688819-edc9-406a-b5da-050ac072cd43,claim=StandardContentClaim
    [resourceClaim=StandardResourceClaim[id=1557687270864-64540,
    container=default, section=28], offset=81777,
    length=280],offset=0,name=done,size=280] transfer relationship not specified
    org.apache.nifi.processor.exception.FlowFileHandlingException:
    StandardFlowFileRecord[uuid=e2688819-edc9-406a-b5da-050ac072cd43,claim=StandardContentClaim
    [resourceClaim=StandardResourceClaim[id=1557687270864-64540,
    container=default, section=28], offset=81777,
    length=280],offset=0,name=done,size=280] transfer relationship not specified
    at
    org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:264)
    at
    org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:335)
    at
    org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:115)
    at
    org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106)
    at
    org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:285)
    at
    org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
    at
    org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
    at
    org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    
    
    Thanks
    David
    

Mime
View raw message