sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gwen Shapira <gshap...@cloudera.com>
Subject Re: Sqoop Import with Postgresql and Kerberos
Date Fri, 21 Nov 2014 09:21:38 GMT
I'd also check /etc/krb5.keytab, you may be running into this issue:
https://bugzilla.redhat.com/show_bug.cgi?id=748528

(It also happens on Ubuntu)

On Fri, Nov 21, 2014 at 12:54 AM, Abraham Elmahrek <abe@cloudera.com> wrote:

> Two things I think might be going on:
>
>    1. kerberosServerName may need to be just the short name "postgre".
>    Not sure.
>    2. jaasApplicationName should be set to "Krb5LoginModule". As per
>    https://docs.oracle.com/javase/7/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5LoginModule.html
>    and http://jdbc.postgresql.org/documentation/84/connect.html.
>
> -Abe
>
> On Thu, Nov 20, 2014 at 5:10 PM, David Lyle <dlyle65535@gmail.com> wrote:
>
>> Thanks for hanging with me. Still same result. Postgres says:
>>
>> FATAL: GSSAPI authentication failed for user "dlyle"
>>
>> Here's the current state of the world:
>>
>> klist -f
>> Ticket cache: FILE:/tmp/krb5cc_115000008
>> Default principal: dlyle@LYLECO.COM
>>
>> Valid starting     Expires            Service principal
>> 11/20/14 20:07:20  11/21/14 20:07:19  krbtgt/LYLECO.COM@LYLECO.COM
>> renew until 11/25/14 20:07:09, Flags: FPRIAT
>>
>> [dlyle@hwx1 ~]$ kinit -R
>> [dlyle@hwx1 ~]$ sqoop list-databases --connect
>> "jdbc:postgresql://
>> postgres.lyleco.com:5432?kerberosServerName=postgres/postgres.lylecom.com@LYLECO.COM
>> "
>> --verbose
>> kinit: Generic preauthentication failure while getting initial credentials
>> kinit: Generic preauthentication failure while getting initial credentials
>> 14/11/20 20:07:59 INFO sqoop.Sqoop: Running Sqoop version:
>> 1.4.4.2.1.7.0-784
>> 14/11/20 20:07:59 DEBUG tool.BaseSqoopTool: Enabled debug logging.
>> 14/11/20 20:07:59 DEBUG sqoop.ConnFactory: Loaded manager factory:
>> com.cloudera.sqoop.manager.DefaultManagerFactory
>> 14/11/20 20:07:59 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
>> com.cloudera.sqoop.manager.DefaultManagerFactory
>> 14/11/20 20:07:59 DEBUG manager.DefaultManagerFactory: Trying with
>> scheme: jdbc:postgresql:
>> 14/11/20 20:07:59 INFO manager.SqlManager: Using default fetchSize of 1000
>> 14/11/20 20:07:59 DEBUG sqoop.ConnFactory: Instantiated ConnManager
>> org.apache.sqoop.manager.PostgresqlManager@5cb42b
>> 14/11/20 20:07:59 DEBUG manager.SqlManager: No connection paramenters
>> specified. Using regular API for making connection.
>> 14/11/20 20:07:59 ERROR manager.CatalogQueryManager: Failed to list
>> databases
>> org.postgresql.util.PSQLException: GSS Authentication failed
>> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:49)
>> at
>> org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:380)
>> at
>> org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
>> at
>> org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
>> at
>> org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
>> at
>> org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
>> at
>> org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
>> at
>> org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
>> at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
>> at org.postgresql.Driver.makeConnection(Driver.java:393)
>> at org.postgresql.Driver.connect(Driver.java:267)
>> at java.sql.DriverManager.getConnection(DriverManager.java:571)
>> at java.sql.DriverManager.getConnection(DriverManager.java:233)
>> at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:824)
>> at
>> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
>> at
>> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
>> at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
>> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
>> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
>> Caused by: javax.security.auth.login.LoginException: No LoginModules
>> configured for pgjdbc
>> at javax.security.auth.login.LoginContext.init(LoginContext.java:273)
>> at javax.security.auth.login.LoginContext.<init>(LoginContext.java:418)
>> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:42)
>> ... 22 more
>> 14/11/20 20:07:59 ERROR sqoop.Sqoop: Got exception running Sqoop:
>> java.lang.RuntimeException: org.postgresql.util.PSQLException: GSS
>> Authentication failed
>> java.lang.RuntimeException: org.postgresql.util.PSQLException: GSS
>> Authentication failed
>> at
>> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:73)
>> at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
>> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
>> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
>> Caused by: org.postgresql.util.PSQLException: GSS Authentication failed
>> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:49)
>> at
>> org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:380)
>> at
>> org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
>> at
>> org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
>> at
>> org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
>> at
>> org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
>> at
>> org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
>> at
>> org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
>> at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
>> at org.postgresql.Driver.makeConnection(Driver.java:393)
>> at org.postgresql.Driver.connect(Driver.java:267)
>> at java.sql.DriverManager.getConnection(DriverManager.java:571)
>> at java.sql.DriverManager.getConnection(DriverManager.java:233)
>> at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:824)
>> at
>> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
>> at
>> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
>> ... 7 more
>> Caused by: javax.security.auth.login.LoginException: No LoginModules
>> configured for pgjdbc
>> at javax.security.auth.login.LoginContext.init(LoginContext.java:273)
>> at javax.security.auth.login.LoginContext.<init>(LoginContext.java:418)
>> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:42)
>> ... 22 more
>>
>> On Thu, Nov 20, 2014 at 7:24 PM, Abraham Elmahrek <abe@cloudera.com>
>> wrote:
>> > Hey there,
>> >
>> > My mistake. Apparently the it's a bit more like an HTTP string:
>> > jdbc:postgresql://
>> postgres.lyleco.com:5432?kerberosServerName=postgres/postgres.lyleco.com@LYLECO.COM
>> .
>> > Also, are the JDBC jars in /var/lib/sqoop? Or are they part of Sqoop's
>> class
>> > path somehow?
>> >
>> > The "Generic preauthentication" errors are discomforting. Are your
>> > credentials renewable? I believe hadoop-auth starts a renewer thread
>> that
>> > runs "kinit -R". That could be causing that error.
>> >
>> > -Abe
>> >
>> > On Thu, Nov 20, 2014 at 3:56 PM, David Lyle <dlyle65535@gmail.com>
>> wrote:
>> >>
>> >> Hi Abe,
>> >>
>> >> Thanks for the quick reply. Currently, the postgres instance in all
>> >> that is kerberized. When I add the kerberosServerName I get:
>> >>
>> >> java.sql.SQLException: No suitable driver found for
>> >>
>> >> jdbc:postgresql://postgres.lyleco.com:5432
>> ;kerberosServerName=postgres/postgres.lyleco.com@LYLECO.COM
>> >>
>> >> or
>> >>
>> >> java.sql.SQLException: No suitable driver found for
>> >>
>> >> jdbc:postgresql://postgres.lyleco.com:5432
>> &kerberosServerName=postgres/postgres.lyleco.com@LYLECO.COM
>> >>
>> >> (I tried both). I'm also troubled by the repeated kinit: Generic
>> >> preauthentication failure while getting initial credentials at the
>> >> very top. I don't understand how sqoop is delegating, but the kinit is
>> >> surprising.
>> >>
>> >> -D...
>> >>
>> >>
>> >> On Thu, Nov 20, 2014 at 6:15 PM, Abraham Elmahrek <abe@cloudera.com>
>> >> wrote:
>> >> > This is a long shot... but I thought I'd say something.
>> >> >
>> >> > If you have Hadoop configured for Kerberos, then Sqoop should
>> >> > automatically
>> >> > use your kerberos credentials to run a MapReduce job. That MapReduce
>> job
>> >> > will run with its own Kerberos principal (normally yarn or mapred).
>> This
>> >> > means that Hadoop has access to postgresql as long as it requests a
>> >> > session
>> >> > from the KDC, assuming postgresql and Hadoop are part of the same
>> realm.
>> >> > My
>> >> > understanding is the JDBC driver will handle this for you if you
>> provide
>> >> > the
>> >> > postgresql service name via "kerberosServerName" in the JDBC
>> connection
>> >> > string:
>> >> >
>> >> > "jdbc:postgresql://postgres.lyleco.com:5432
>> ;kerberosServerName=postgres/<host>@<REALM>".
>> >> >
>> >> > Ref: http://jdbc.postgresql.org/documentation/84/connect.html
>> >> >
>> >> > -Abe
>> >> >
>> >> > On Thu, Nov 20, 2014 at 2:53 PM, David Lyle <dlyle65535@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> I'm trying to use Sqoop to connect to a kerberized Postgres
>> instance.
>> >> >> At this time, I can connect from psql, but my sqoop session gives
me
>> >> >> the log below. Any idea where I'm going wrong?
>> >> >>
>> >> >> Thanks!
>> >> >>
>> >> >> -David...
>> >> >>
>> >> >>
>> >> >> sqoop list-databases  -Djavax.security.auth.debug=true --connect
>> >> >> "jdbc:postgresql://postgres.lyleco.com:5432" -verbose
>> >> >>
>> >> >> kinit: Generic preauthentication failure while getting initial
>> >> >> credentials
>> >> >> kinit: Generic preauthentication failure while getting initial
>> >> >> credentials
>> >> >>
>> >> >> 14/11/20 17:26:37 INFO sqoop.Sqoop: Running Sqoop version:
>> >> >> 1.4.4.2.1.7.0-784
>> >> >> 14/11/20 17:26:37 DEBUG tool.BaseSqoopTool: Enabled debug logging.
>> >> >> 14/11/20 17:26:37 DEBUG sqoop.ConnFactory: Loaded manager factory:
>> >> >> com.cloudera.sqoop.manager.DefaultManagerFactory
>> >> >> 14/11/20 17:26:37 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
>> >> >> com.cloudera.sqoop.manager.DefaultManagerFactory
>> >> >> 14/11/20 17:26:37 DEBUG manager.DefaultManagerFactory: Trying with
>> >> >> scheme: jdbc:postgresql:
>> >> >> 14/11/20 17:26:37 INFO manager.SqlManager: Using default fetchSize
>> of
>> >> >> 1000
>> >> >> 14/11/20 17:26:37 DEBUG sqoop.ConnFactory: Instantiated ConnManager
>> >> >> org.apache.sqoop.manager.PostgresqlManager@1663108
>> >> >> 14/11/20 17:26:37 DEBUG manager.SqlManager: No connection
>> paramenters
>> >> >> specified. Using regular API for making connection.
>> >> >> 14/11/20 17:26:37 ERROR manager.CatalogQueryManager: Failed to
list
>> >> >> databases
>> >> >> org.postgresql.util.PSQLException: GSS Authentication failed
>> >> >> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:49)
>> >> >> at
>> >> >>
>> >> >>
>> org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:380)
>> >> >> at
>> >> >>
>> >> >>
>> org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
>> >> >> at
>> >> >>
>> >> >>
>> org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
>> >> >> at
>> >> >>
>> >> >>
>> org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
>> >> >> at
>> >> >>
>> >> >>
>> org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
>> >> >> at
>> >> >>
>> >> >>
>> org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
>> >> >> at
>> >> >>
>> >> >>
>> org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
>> >> >> at
>> org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
>> >> >> at org.postgresql.Driver.makeConnection(Driver.java:393)
>> >> >> at org.postgresql.Driver.connect(Driver.java:267)
>> >> >> at java.sql.DriverManager.getConnection(DriverManager.java:571)
>> >> >> at java.sql.DriverManager.getConnection(DriverManager.java:233)
>> >> >> at
>> >> >>
>> org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:824)
>> >> >> at
>> >> >>
>> >> >>
>> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
>> >> >> at
>> >> >>
>> >> >>
>> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
>> >> >> at
>> >> >>
>> org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
>> >> >> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
>> >> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> >> >> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
>> >> >> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
>> >> >> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
>> >> >> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
>> >> >> Caused by: javax.security.auth.login.LoginException: No LoginModules
>> >> >> configured for pgjdbc
>> >> >> at
>> javax.security.auth.login.LoginContext.init(LoginContext.java:273)
>> >> >> at
>> javax.security.auth.login.LoginContext.<init>(LoginContext.java:418)
>> >> >> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:42)
>> >> >> ... 22 more
>> >> >> 14/11/20 17:26:37 ERROR sqoop.Sqoop: Got exception running Sqoop:
>> >> >> java.lang.RuntimeException: org.postgresql.util.PSQLException:
GSS
>> >> >> Authentication failed
>> >> >> java.lang.RuntimeException: org.postgresql.util.PSQLException:
GSS
>> >> >> Authentication failed
>> >> >> at
>> >> >>
>> >> >>
>> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:73)
>> >> >> at
>> >> >>
>> org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
>> >> >> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
>> >> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> >> >> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
>> >> >> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
>> >> >> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
>> >> >> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
>> >> >> Caused by: org.postgresql.util.PSQLException: GSS Authentication
>> failed
>> >> >> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:49)
>> >> >> at
>> >> >>
>> >> >>
>> org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:380)
>> >> >> at
>> >> >>
>> >> >>
>> org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
>> >> >> at
>> >> >>
>> >> >>
>> org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
>> >> >> at
>> >> >>
>> >> >>
>> org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
>> >> >> at
>> >> >>
>> >> >>
>> org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
>> >> >> at
>> >> >>
>> >> >>
>> org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
>> >> >> at
>> >> >>
>> >> >>
>> org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
>> >> >> at
>> org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
>> >> >> at org.postgresql.Driver.makeConnection(Driver.java:393)
>> >> >> at org.postgresql.Driver.connect(Driver.java:267)
>> >> >> at java.sql.DriverManager.getConnection(DriverManager.java:571)
>> >> >> at java.sql.DriverManager.getConnection(DriverManager.java:233)
>> >> >> at
>> >> >>
>> org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:824)
>> >> >> at
>> >> >>
>> >> >>
>> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
>> >> >> at
>> >> >>
>> >> >>
>> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
>> >> >> ... 7 more
>> >> >> Caused by: javax.security.auth.login.LoginException: No LoginModules
>> >> >> configured for pgjdbc
>> >> >> at
>> javax.security.auth.login.LoginContext.init(LoginContext.java:273)
>> >> >> at
>> javax.security.auth.login.LoginContext.<init>(LoginContext.java:418)
>> >> >> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:42)
>> >> >> ... 22 more
>> >> >
>> >> >
>> >
>> >
>>
>
>

Mime
View raw message