nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Giannone <dgiann...@humana.com>
Subject RE: SelectHiveQL Error
Date Fri, 07 Oct 2016 13:49:13 GMT
It turns out the port needed to be changed for hive server2 as well. That seemed to fix the
below issue. However, now I get :

> org.apache.thrift.TApplicationException: Required field 'client_protocol' is unset!

Which according to this<http://stackoverflow.com/questions/30931599/error-jdbc-hiveconnection-error-opening-session-hive>
indicates my hive and hive-jdbc versions are mismatching. “Hive –-version” gives me
1.1.0. If I were to download the hive-jdbc 1.1.0 jar, is there a way I could specify that
it us that?


-Dan

From: Dan Giannone [mailto:dgiannone@humana.com]
Sent: Friday, October 07, 2016 9:25 AM
To: users@nifi.apache.org
Subject: RE: SelectHiveQL Error

Hi Matt,

When I try to change to jdbc:hive2://, I get a different error set of errors.

>Error getting Hive connection
>org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could
not open client transport with JDBC Uri: jdbc:hive2://…)
>Caused by: java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://…
>Caused by: org.apache.thrift.transport.TTransportException: null

I am thinking you are right in that it is an issue with my connection URL. Is there some command
I can run that will generate this for me? Or a specific place I should look? The only mention
of a url in hive-site.xml that I see is:

<property>
    <name>hive.metastore.uris</name>
    <value>thrift://server:port</value>
</property>


-Dan

From: Matt Burgess [mailto:mattyb149@gmail.com]
Sent: Thursday, October 06, 2016 5:17 PM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: SelectHiveQL Error

Andrew is correct. Although the HiveServer
1 driver is included with the NAR, the HiveConnectionPool is hardcoded to use the HiveServer
2 driver (since the former doesn't allow for simultaneous connections and we are using a connection
pool :) the scheme should be jdbc:hive2:// not hive.

If that was a typo and you are using the correct scheme, could you provide your configuration
details/properties?

Thanks,
Matt


On Oct 6, 2016, at 4:07 PM, Andrew Grande <aperepel@gmail.com<mailto:aperepel@gmail.com>>
wrote:

Are you sure the jdbc url is correct? Iirc, it was jdbc:hive2://

Andrew

On Thu, Oct 6, 2016, 3:46 PM Dan Giannone <dgiannone@humana.com<mailto:dgiannone@humana.com>>
wrote:
Hi Matt,

Here is the whole error trace, starting from when I turned on the SelectHiveQL processor:

INFO [StandardProcessScheduler Thread-2] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled SelectHiveQL[id=0157102a-94da-11ec-0f7e-17fd3119aa00]
to run with 1 threads
2016-10-06 15:37:06,554 INFO [Timer-Driven Process Thread-7] o.a.nifi.dbcp.hive.HiveConnectionPool
HiveConnectionPool[id=0157102d-94da-11ec-4d91-5a8952e888bd] Simple Authentication
2016-10-06 15:37:06,556 ERROR [Timer-Driven Process Thread-7] o.a.nifi.dbcp.hive.HiveConnectionPool
HiveConnectionPool[id=0157102d-94da-11ec-4d91-5a8952e888bd] Error getting Hive connection
2016-10-06 15:37:06,557 ERROR [Timer-Driven Process Thread-7] o.a.nifi.dbcp.hive.HiveConnectionPool
org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of class 'org.apache.hive.jdbc.HiveDriver'
for connect URL 'jdbc:hive://server:port/default'
        at org.apache.commons.dbcp.BasicDataSource.createConnectionFactory(BasicDataSource.java:1452)
~[commons-dbcp-1.4.jar:1.4]
        at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1371)
~[commons-dbcp-1.4.jar:1.4]
        at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
~[commons-dbcp-1.4.jar:1.4]
        at org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:269)
~[nifi-hive-processors-1.0.0.jar:1.0.0]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_45]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45]
        at org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:177)
[nifi-framework-core-1.0.0.jar:1.0.0]
        at com.sun.proxy.$Proxy81.getConnection(Unknown Source) [na:na]
        at org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:158) [nifi-hive-processors-1.0.0.jar:1.0.0]
        at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
[nifi-api-1.0.0.jar:1.0.0]
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064)
[nifi-framework-core-1.0.0.jar:1.0.0]
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.0.0.jar:1.0.0]
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.0.0.jar:1.0.0]
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
[nifi-framework-core-1.0.0.jar:1.0.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: java.sql.SQLException: No suitable driver
        at java.sql.DriverManager.getDriver(DriverManager.java:315) ~[na:1.8.0_45]
        at org.apache.commons.dbcp.BasicDataSource.createConnectionFactory(BasicDataSource.java:1437)
~[commons-dbcp-1.4.jar:1.4]
        ... 22 common frames omitted
2016-10-06 15:37:06,557 ERROR [Timer-Driven Process Thread-7] o.a.nifi.processors.hive.SelectHiveQL
SelectHiveQL[id=0157102a-94da-11ec-0f7e-17fd3119aa00] Unable to execute HiveQL select query
select * from dan_test due to org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException:
Cannot create JDBC driver of class 'org.apache.hive.jdbc.HiveDriver' for connect URL 'jdbc:hive://server:port/default'.
No FlowFile to route to failure: org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException:
Cannot create JDBC driver of class 'org.apache.hive.jdbc.HiveDriver' for connect URL 'jdbc:hive://server:port/default'
2016-10-06 15:37:06,558 ERROR [Timer-Driven Process Thread-7] o.a.nifi.processors.hive.SelectHiveQL
org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException:
Cannot create JDBC driver of class 'org.apache.hive.jdbc.HiveDriver' for connect URL 'jdbc:hive://server:port/default'
        at org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:273)
~[nifi-hive-processors-1.0.0.jar:1.0.0]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_45]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45]
        at org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:177)
~[na:na]
        at com.sun.proxy.$Proxy81.getConnection(Unknown Source) ~[na:na]
        at org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:158) ~[nifi-hive-processors-1.0.0.jar:1.0.0]
        at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
[nifi-api-1.0.0.jar:1.0.0]
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064)
[nifi-framework-core-1.0.0.jar:1.0.0]
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.0.0.jar:1.0.0]
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.0.0.jar:1.0.0]
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
[nifi-framework-core-1.0.0.jar:1.0.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of class
'org.apache.hive.jdbc.HiveDriver' for connect URL 'jdbc:hive://server:port/default'
        at org.apache.commons.dbcp.BasicDataSource.createConnectionFactory(BasicDataSource.java:1452)
~[commons-dbcp-1.4.jar:1.4]
        at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1371)
~[commons-dbcp-1.4.jar:1.4]
        at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
~[commons-dbcp-1.4.jar:1.4]
        at org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:269)
~[nifi-hive-processors-1.0.0.jar:1.0.0]
        ... 19 common frames omitted
Caused by: java.sql.SQLException: No suitable driver
        at java.sql.DriverManager.getDriver(DriverManager.java:315) ~[na:1.8.0_45]
        at org.apache.commons.dbcp.BasicDataSource.createConnectionFactory(BasicDataSource.java:1437)
~[commons-dbcp-1.4.jar:1.4]
        ... 22 common frames omitted


Sorry about the formatting. Any ideas? Is there some way I can edit the Hive Nar file?

-Dan







-----Original Message-----
From: Matt Burgess [mailto:mattyb149@apache.org<mailto:mattyb149@apache.org>]
Sent: Thursday, October 06, 2016 3:24 PM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: SelectHiveQL Error

Dan,

That is a catch-all error returned when (in case probably) something is misconfigured. Are
there more error lines below that in the log?
The driver class and all its dependencies are present in the Hive NAR, so there is likely
an underlying error that, while being propagated up, returns the generic (and misleading)
error message you describe.

Regards,
Matt

On Thu, Oct 6, 2016 at 3:14 PM, Dan Giannone <dgiannone@humana.com<mailto:dgiannone@humana.com>>
wrote:
> Hello,
>
>
>
> I am trying to use a SelectHiveQL processor using the following
> controller
> configuration:
>
>
>
> Database Connection URL - jdbc:hive://server:port/default
>
> Hive Configuration Resources - /path/to/hive/hive-site.xml
>
>
>
> When my processor goes to run the simple query, I get the following
> error (copying the important parts to save space):
>
>
>
>>Unable to execute HiveQL select query select * from table
>
>>Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create
>>JDBC  driver of class 'org.apache.hive.jdbc.HiveDriver' for connect
>>URL ‘<see
>> above>’
>
>>Caused by: java.sql.SQLException: No suitable driver
>
>
>
> I assume I am missing something in hive-site.xml, or a jar file? Any
> insight would be appreciated.
>
>
>
> Thanks,
>
>
>
> Dan
>
>
> The information transmitted is intended only for the person or entity
> to which it is addressed and may contain CONFIDENTIAL material. If you
> receive this material/information in error, please contact the sender
> and delete or destroy the material/information.

The information transmitted is intended only for the person or entity to which it is addressed
and may contain CONFIDENTIAL material.  If you receive this material/information in error,
please contact the sender and delete or destroy the material/information.

The information transmitted is intended only for the person or entity to which it is addressed
and may contain CONFIDENTIAL material. If you receive this material/information in error,
please contact the sender and delete or destroy the material/information.

The information transmitted is intended only for the person or entity to which it is addressed
and may contain CONFIDENTIAL material.  If you receive this material/information in error,
please contact the sender and delete or destroy the material/information.
Mime
View raw message