sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Lyle <dlyle65...@gmail.com>
Subject Re: Sqoop Import with Postgresql and Kerberos
Date Fri, 21 Nov 2014 01:10:38 GMT
Thanks for hanging with me. Still same result. Postgres says:

FATAL: GSSAPI authentication failed for user "dlyle"

Here's the current state of the world:

klist -f
Ticket cache: FILE:/tmp/krb5cc_115000008
Default principal: dlyle@LYLECO.COM

Valid starting     Expires            Service principal
11/20/14 20:07:20  11/21/14 20:07:19  krbtgt/LYLECO.COM@LYLECO.COM
renew until 11/25/14 20:07:09, Flags: FPRIAT

[dlyle@hwx1 ~]$ kinit -R
[dlyle@hwx1 ~]$ sqoop list-databases --connect
"jdbc:postgresql://postgres.lyleco.com:5432?kerberosServerName=postgres/postgres.lylecom.com@LYLECO.COM"
--verbose
kinit: Generic preauthentication failure while getting initial credentials
kinit: Generic preauthentication failure while getting initial credentials
14/11/20 20:07:59 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4.2.1.7.0-784
14/11/20 20:07:59 DEBUG tool.BaseSqoopTool: Enabled debug logging.
14/11/20 20:07:59 DEBUG sqoop.ConnFactory: Loaded manager factory:
com.cloudera.sqoop.manager.DefaultManagerFactory
14/11/20 20:07:59 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
com.cloudera.sqoop.manager.DefaultManagerFactory
14/11/20 20:07:59 DEBUG manager.DefaultManagerFactory: Trying with
scheme: jdbc:postgresql:
14/11/20 20:07:59 INFO manager.SqlManager: Using default fetchSize of 1000
14/11/20 20:07:59 DEBUG sqoop.ConnFactory: Instantiated ConnManager
org.apache.sqoop.manager.PostgresqlManager@5cb42b
14/11/20 20:07:59 DEBUG manager.SqlManager: No connection paramenters
specified. Using regular API for making connection.
14/11/20 20:07:59 ERROR manager.CatalogQueryManager: Failed to list databases
org.postgresql.util.PSQLException: GSS Authentication failed
at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:49)
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:380)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
at org.postgresql.Driver.makeConnection(Driver.java:393)
at org.postgresql.Driver.connect(Driver.java:267)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:233)
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:824)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
Caused by: javax.security.auth.login.LoginException: No LoginModules
configured for pgjdbc
at javax.security.auth.login.LoginContext.init(LoginContext.java:273)
at javax.security.auth.login.LoginContext.<init>(LoginContext.java:418)
at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:42)
... 22 more
14/11/20 20:07:59 ERROR sqoop.Sqoop: Got exception running Sqoop:
java.lang.RuntimeException: org.postgresql.util.PSQLException: GSS
Authentication failed
java.lang.RuntimeException: org.postgresql.util.PSQLException: GSS
Authentication failed
at org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:73)
at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
Caused by: org.postgresql.util.PSQLException: GSS Authentication failed
at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:49)
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:380)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
at org.postgresql.Driver.makeConnection(Driver.java:393)
at org.postgresql.Driver.connect(Driver.java:267)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:233)
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:824)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
... 7 more
Caused by: javax.security.auth.login.LoginException: No LoginModules
configured for pgjdbc
at javax.security.auth.login.LoginContext.init(LoginContext.java:273)
at javax.security.auth.login.LoginContext.<init>(LoginContext.java:418)
at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:42)
... 22 more

On Thu, Nov 20, 2014 at 7:24 PM, Abraham Elmahrek <abe@cloudera.com> wrote:
> Hey there,
>
> My mistake. Apparently the it's a bit more like an HTTP string:
> jdbc:postgresql://postgres.lyleco.com:5432?kerberosServerName=postgres/postgres.lyleco.com@LYLECO.COM.
> Also, are the JDBC jars in /var/lib/sqoop? Or are they part of Sqoop's class
> path somehow?
>
> The "Generic preauthentication" errors are discomforting. Are your
> credentials renewable? I believe hadoop-auth starts a renewer thread that
> runs "kinit -R". That could be causing that error.
>
> -Abe
>
> On Thu, Nov 20, 2014 at 3:56 PM, David Lyle <dlyle65535@gmail.com> wrote:
>>
>> Hi Abe,
>>
>> Thanks for the quick reply. Currently, the postgres instance in all
>> that is kerberized. When I add the kerberosServerName I get:
>>
>> java.sql.SQLException: No suitable driver found for
>>
>> jdbc:postgresql://postgres.lyleco.com:5432;kerberosServerName=postgres/postgres.lyleco.com@LYLECO.COM
>>
>> or
>>
>> java.sql.SQLException: No suitable driver found for
>>
>> jdbc:postgresql://postgres.lyleco.com:5432&kerberosServerName=postgres/postgres.lyleco.com@LYLECO.COM
>>
>> (I tried both). I'm also troubled by the repeated kinit: Generic
>> preauthentication failure while getting initial credentials at the
>> very top. I don't understand how sqoop is delegating, but the kinit is
>> surprising.
>>
>> -D...
>>
>>
>> On Thu, Nov 20, 2014 at 6:15 PM, Abraham Elmahrek <abe@cloudera.com>
>> wrote:
>> > This is a long shot... but I thought I'd say something.
>> >
>> > If you have Hadoop configured for Kerberos, then Sqoop should
>> > automatically
>> > use your kerberos credentials to run a MapReduce job. That MapReduce job
>> > will run with its own Kerberos principal (normally yarn or mapred). This
>> > means that Hadoop has access to postgresql as long as it requests a
>> > session
>> > from the KDC, assuming postgresql and Hadoop are part of the same realm.
>> > My
>> > understanding is the JDBC driver will handle this for you if you provide
>> > the
>> > postgresql service name via "kerberosServerName" in the JDBC connection
>> > string:
>> >
>> > "jdbc:postgresql://postgres.lyleco.com:5432;kerberosServerName=postgres/<host>@<REALM>".
>> >
>> > Ref: http://jdbc.postgresql.org/documentation/84/connect.html
>> >
>> > -Abe
>> >
>> > On Thu, Nov 20, 2014 at 2:53 PM, David Lyle <dlyle65535@gmail.com>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> I'm trying to use Sqoop to connect to a kerberized Postgres instance.
>> >> At this time, I can connect from psql, but my sqoop session gives me
>> >> the log below. Any idea where I'm going wrong?
>> >>
>> >> Thanks!
>> >>
>> >> -David...
>> >>
>> >>
>> >> sqoop list-databases  -Djavax.security.auth.debug=true --connect
>> >> "jdbc:postgresql://postgres.lyleco.com:5432" -verbose
>> >>
>> >> kinit: Generic preauthentication failure while getting initial
>> >> credentials
>> >> kinit: Generic preauthentication failure while getting initial
>> >> credentials
>> >>
>> >> 14/11/20 17:26:37 INFO sqoop.Sqoop: Running Sqoop version:
>> >> 1.4.4.2.1.7.0-784
>> >> 14/11/20 17:26:37 DEBUG tool.BaseSqoopTool: Enabled debug logging.
>> >> 14/11/20 17:26:37 DEBUG sqoop.ConnFactory: Loaded manager factory:
>> >> com.cloudera.sqoop.manager.DefaultManagerFactory
>> >> 14/11/20 17:26:37 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
>> >> com.cloudera.sqoop.manager.DefaultManagerFactory
>> >> 14/11/20 17:26:37 DEBUG manager.DefaultManagerFactory: Trying with
>> >> scheme: jdbc:postgresql:
>> >> 14/11/20 17:26:37 INFO manager.SqlManager: Using default fetchSize of
>> >> 1000
>> >> 14/11/20 17:26:37 DEBUG sqoop.ConnFactory: Instantiated ConnManager
>> >> org.apache.sqoop.manager.PostgresqlManager@1663108
>> >> 14/11/20 17:26:37 DEBUG manager.SqlManager: No connection paramenters
>> >> specified. Using regular API for making connection.
>> >> 14/11/20 17:26:37 ERROR manager.CatalogQueryManager: Failed to list
>> >> databases
>> >> org.postgresql.util.PSQLException: GSS Authentication failed
>> >> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:49)
>> >> at
>> >>
>> >> org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:380)
>> >> at
>> >>
>> >> org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
>> >> at
>> >>
>> >> org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
>> >> at
>> >>
>> >> org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
>> >> at
>> >>
>> >> org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
>> >> at
>> >>
>> >> org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
>> >> at
>> >>
>> >> org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
>> >> at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
>> >> at org.postgresql.Driver.makeConnection(Driver.java:393)
>> >> at org.postgresql.Driver.connect(Driver.java:267)
>> >> at java.sql.DriverManager.getConnection(DriverManager.java:571)
>> >> at java.sql.DriverManager.getConnection(DriverManager.java:233)
>> >> at
>> >> org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:824)
>> >> at
>> >>
>> >> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
>> >> at
>> >>
>> >> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
>> >> at
>> >> org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
>> >> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
>> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> >> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
>> >> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
>> >> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
>> >> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
>> >> Caused by: javax.security.auth.login.LoginException: No LoginModules
>> >> configured for pgjdbc
>> >> at javax.security.auth.login.LoginContext.init(LoginContext.java:273)
>> >> at javax.security.auth.login.LoginContext.<init>(LoginContext.java:418)
>> >> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:42)
>> >> ... 22 more
>> >> 14/11/20 17:26:37 ERROR sqoop.Sqoop: Got exception running Sqoop:
>> >> java.lang.RuntimeException: org.postgresql.util.PSQLException: GSS
>> >> Authentication failed
>> >> java.lang.RuntimeException: org.postgresql.util.PSQLException: GSS
>> >> Authentication failed
>> >> at
>> >>
>> >> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:73)
>> >> at
>> >> org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
>> >> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
>> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> >> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
>> >> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
>> >> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
>> >> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
>> >> Caused by: org.postgresql.util.PSQLException: GSS Authentication failed
>> >> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:49)
>> >> at
>> >>
>> >> org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:380)
>> >> at
>> >>
>> >> org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
>> >> at
>> >>
>> >> org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
>> >> at
>> >>
>> >> org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
>> >> at
>> >>
>> >> org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
>> >> at
>> >>
>> >> org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
>> >> at
>> >>
>> >> org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
>> >> at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
>> >> at org.postgresql.Driver.makeConnection(Driver.java:393)
>> >> at org.postgresql.Driver.connect(Driver.java:267)
>> >> at java.sql.DriverManager.getConnection(DriverManager.java:571)
>> >> at java.sql.DriverManager.getConnection(DriverManager.java:233)
>> >> at
>> >> org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:824)
>> >> at
>> >>
>> >> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
>> >> at
>> >>
>> >> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
>> >> ... 7 more
>> >> Caused by: javax.security.auth.login.LoginException: No LoginModules
>> >> configured for pgjdbc
>> >> at javax.security.auth.login.LoginContext.init(LoginContext.java:273)
>> >> at javax.security.auth.login.LoginContext.<init>(LoginContext.java:418)
>> >> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:42)
>> >> ... 22 more
>> >
>> >
>
>

Mime
View raw message