sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abraham Elmahrek <...@cloudera.com>
Subject Re: Sqoop Import with Postgresql and Kerberos
Date Fri, 21 Nov 2014 08:54:16 GMT
Two things I think might be going on:

   1. kerberosServerName may need to be just the short name "postgre". Not
   sure.
   2. jaasApplicationName should be set to "Krb5LoginModule". As per
   https://docs.oracle.com/javase/7/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5LoginModule.html
   and http://jdbc.postgresql.org/documentation/84/connect.html.

-Abe

On Thu, Nov 20, 2014 at 5:10 PM, David Lyle <dlyle65535@gmail.com> wrote:

> Thanks for hanging with me. Still same result. Postgres says:
>
> FATAL: GSSAPI authentication failed for user "dlyle"
>
> Here's the current state of the world:
>
> klist -f
> Ticket cache: FILE:/tmp/krb5cc_115000008
> Default principal: dlyle@LYLECO.COM
>
> Valid starting     Expires            Service principal
> 11/20/14 20:07:20  11/21/14 20:07:19  krbtgt/LYLECO.COM@LYLECO.COM
> renew until 11/25/14 20:07:09, Flags: FPRIAT
>
> [dlyle@hwx1 ~]$ kinit -R
> [dlyle@hwx1 ~]$ sqoop list-databases --connect
> "jdbc:postgresql://
> postgres.lyleco.com:5432?kerberosServerName=postgres/postgres.lylecom.com@LYLECO.COM
> "
> --verbose
> kinit: Generic preauthentication failure while getting initial credentials
> kinit: Generic preauthentication failure while getting initial credentials
> 14/11/20 20:07:59 INFO sqoop.Sqoop: Running Sqoop version:
> 1.4.4.2.1.7.0-784
> 14/11/20 20:07:59 DEBUG tool.BaseSqoopTool: Enabled debug logging.
> 14/11/20 20:07:59 DEBUG sqoop.ConnFactory: Loaded manager factory:
> com.cloudera.sqoop.manager.DefaultManagerFactory
> 14/11/20 20:07:59 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
> com.cloudera.sqoop.manager.DefaultManagerFactory
> 14/11/20 20:07:59 DEBUG manager.DefaultManagerFactory: Trying with
> scheme: jdbc:postgresql:
> 14/11/20 20:07:59 INFO manager.SqlManager: Using default fetchSize of 1000
> 14/11/20 20:07:59 DEBUG sqoop.ConnFactory: Instantiated ConnManager
> org.apache.sqoop.manager.PostgresqlManager@5cb42b
> 14/11/20 20:07:59 DEBUG manager.SqlManager: No connection paramenters
> specified. Using regular API for making connection.
> 14/11/20 20:07:59 ERROR manager.CatalogQueryManager: Failed to list
> databases
> org.postgresql.util.PSQLException: GSS Authentication failed
> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:49)
> at
> org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:380)
> at
> org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
> at
> org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
> at
> org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
> at
> org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
> at
> org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
> at
> org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
> at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
> at org.postgresql.Driver.makeConnection(Driver.java:393)
> at org.postgresql.Driver.connect(Driver.java:267)
> at java.sql.DriverManager.getConnection(DriverManager.java:571)
> at java.sql.DriverManager.getConnection(DriverManager.java:233)
> at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:824)
> at
> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
> at
> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
> at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
> Caused by: javax.security.auth.login.LoginException: No LoginModules
> configured for pgjdbc
> at javax.security.auth.login.LoginContext.init(LoginContext.java:273)
> at javax.security.auth.login.LoginContext.<init>(LoginContext.java:418)
> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:42)
> ... 22 more
> 14/11/20 20:07:59 ERROR sqoop.Sqoop: Got exception running Sqoop:
> java.lang.RuntimeException: org.postgresql.util.PSQLException: GSS
> Authentication failed
> java.lang.RuntimeException: org.postgresql.util.PSQLException: GSS
> Authentication failed
> at
> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:73)
> at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
> Caused by: org.postgresql.util.PSQLException: GSS Authentication failed
> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:49)
> at
> org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:380)
> at
> org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
> at
> org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
> at
> org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
> at
> org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
> at
> org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
> at
> org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
> at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
> at org.postgresql.Driver.makeConnection(Driver.java:393)
> at org.postgresql.Driver.connect(Driver.java:267)
> at java.sql.DriverManager.getConnection(DriverManager.java:571)
> at java.sql.DriverManager.getConnection(DriverManager.java:233)
> at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:824)
> at
> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
> at
> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
> ... 7 more
> Caused by: javax.security.auth.login.LoginException: No LoginModules
> configured for pgjdbc
> at javax.security.auth.login.LoginContext.init(LoginContext.java:273)
> at javax.security.auth.login.LoginContext.<init>(LoginContext.java:418)
> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:42)
> ... 22 more
>
> On Thu, Nov 20, 2014 at 7:24 PM, Abraham Elmahrek <abe@cloudera.com>
> wrote:
> > Hey there,
> >
> > My mistake. Apparently the it's a bit more like an HTTP string:
> > jdbc:postgresql://
> postgres.lyleco.com:5432?kerberosServerName=postgres/postgres.lyleco.com@LYLECO.COM
> .
> > Also, are the JDBC jars in /var/lib/sqoop? Or are they part of Sqoop's
> class
> > path somehow?
> >
> > The "Generic preauthentication" errors are discomforting. Are your
> > credentials renewable? I believe hadoop-auth starts a renewer thread that
> > runs "kinit -R". That could be causing that error.
> >
> > -Abe
> >
> > On Thu, Nov 20, 2014 at 3:56 PM, David Lyle <dlyle65535@gmail.com>
> wrote:
> >>
> >> Hi Abe,
> >>
> >> Thanks for the quick reply. Currently, the postgres instance in all
> >> that is kerberized. When I add the kerberosServerName I get:
> >>
> >> java.sql.SQLException: No suitable driver found for
> >>
> >> jdbc:postgresql://postgres.lyleco.com:5432;kerberosServerName=postgres/
> postgres.lyleco.com@LYLECO.COM
> >>
> >> or
> >>
> >> java.sql.SQLException: No suitable driver found for
> >>
> >> jdbc:postgresql://postgres.lyleco.com:5432&kerberosServerName=postgres/
> postgres.lyleco.com@LYLECO.COM
> >>
> >> (I tried both). I'm also troubled by the repeated kinit: Generic
> >> preauthentication failure while getting initial credentials at the
> >> very top. I don't understand how sqoop is delegating, but the kinit is
> >> surprising.
> >>
> >> -D...
> >>
> >>
> >> On Thu, Nov 20, 2014 at 6:15 PM, Abraham Elmahrek <abe@cloudera.com>
> >> wrote:
> >> > This is a long shot... but I thought I'd say something.
> >> >
> >> > If you have Hadoop configured for Kerberos, then Sqoop should
> >> > automatically
> >> > use your kerberos credentials to run a MapReduce job. That MapReduce
> job
> >> > will run with its own Kerberos principal (normally yarn or mapred).
> This
> >> > means that Hadoop has access to postgresql as long as it requests a
> >> > session
> >> > from the KDC, assuming postgresql and Hadoop are part of the same
> realm.
> >> > My
> >> > understanding is the JDBC driver will handle this for you if you
> provide
> >> > the
> >> > postgresql service name via "kerberosServerName" in the JDBC
> connection
> >> > string:
> >> >
> >> > "jdbc:postgresql://postgres.lyleco.com:5432
> ;kerberosServerName=postgres/<host>@<REALM>".
> >> >
> >> > Ref: http://jdbc.postgresql.org/documentation/84/connect.html
> >> >
> >> > -Abe
> >> >
> >> > On Thu, Nov 20, 2014 at 2:53 PM, David Lyle <dlyle65535@gmail.com>
> >> > wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> I'm trying to use Sqoop to connect to a kerberized Postgres instance.
> >> >> At this time, I can connect from psql, but my sqoop session gives me
> >> >> the log below. Any idea where I'm going wrong?
> >> >>
> >> >> Thanks!
> >> >>
> >> >> -David...
> >> >>
> >> >>
> >> >> sqoop list-databases  -Djavax.security.auth.debug=true --connect
> >> >> "jdbc:postgresql://postgres.lyleco.com:5432" -verbose
> >> >>
> >> >> kinit: Generic preauthentication failure while getting initial
> >> >> credentials
> >> >> kinit: Generic preauthentication failure while getting initial
> >> >> credentials
> >> >>
> >> >> 14/11/20 17:26:37 INFO sqoop.Sqoop: Running Sqoop version:
> >> >> 1.4.4.2.1.7.0-784
> >> >> 14/11/20 17:26:37 DEBUG tool.BaseSqoopTool: Enabled debug logging.
> >> >> 14/11/20 17:26:37 DEBUG sqoop.ConnFactory: Loaded manager factory:
> >> >> com.cloudera.sqoop.manager.DefaultManagerFactory
> >> >> 14/11/20 17:26:37 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
> >> >> com.cloudera.sqoop.manager.DefaultManagerFactory
> >> >> 14/11/20 17:26:37 DEBUG manager.DefaultManagerFactory: Trying with
> >> >> scheme: jdbc:postgresql:
> >> >> 14/11/20 17:26:37 INFO manager.SqlManager: Using default fetchSize
of
> >> >> 1000
> >> >> 14/11/20 17:26:37 DEBUG sqoop.ConnFactory: Instantiated ConnManager
> >> >> org.apache.sqoop.manager.PostgresqlManager@1663108
> >> >> 14/11/20 17:26:37 DEBUG manager.SqlManager: No connection paramenters
> >> >> specified. Using regular API for making connection.
> >> >> 14/11/20 17:26:37 ERROR manager.CatalogQueryManager: Failed to list
> >> >> databases
> >> >> org.postgresql.util.PSQLException: GSS Authentication failed
> >> >> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:49)
> >> >> at
> >> >>
> >> >>
> org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:380)
> >> >> at
> >> >>
> >> >>
> org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
> >> >> at
> >> >>
> >> >>
> org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
> >> >> at
> >> >>
> >> >>
> org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
> >> >> at
> >> >>
> >> >>
> org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
> >> >> at
> >> >>
> >> >>
> org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
> >> >> at
> >> >>
> >> >>
> org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
> >> >> at
> org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
> >> >> at org.postgresql.Driver.makeConnection(Driver.java:393)
> >> >> at org.postgresql.Driver.connect(Driver.java:267)
> >> >> at java.sql.DriverManager.getConnection(DriverManager.java:571)
> >> >> at java.sql.DriverManager.getConnection(DriverManager.java:233)
> >> >> at
> >> >>
> org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:824)
> >> >> at
> >> >>
> >> >>
> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
> >> >> at
> >> >>
> >> >>
> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
> >> >> at
> >> >>
> org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
> >> >> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
> >> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >> >> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> >> >> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
> >> >> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
> >> >> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
> >> >> Caused by: javax.security.auth.login.LoginException: No LoginModules
> >> >> configured for pgjdbc
> >> >> at javax.security.auth.login.LoginContext.init(LoginContext.java:273)
> >> >> at
> javax.security.auth.login.LoginContext.<init>(LoginContext.java:418)
> >> >> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:42)
> >> >> ... 22 more
> >> >> 14/11/20 17:26:37 ERROR sqoop.Sqoop: Got exception running Sqoop:
> >> >> java.lang.RuntimeException: org.postgresql.util.PSQLException: GSS
> >> >> Authentication failed
> >> >> java.lang.RuntimeException: org.postgresql.util.PSQLException: GSS
> >> >> Authentication failed
> >> >> at
> >> >>
> >> >>
> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:73)
> >> >> at
> >> >>
> org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
> >> >> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
> >> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >> >> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> >> >> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
> >> >> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
> >> >> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
> >> >> Caused by: org.postgresql.util.PSQLException: GSS Authentication
> failed
> >> >> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:49)
> >> >> at
> >> >>
> >> >>
> org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:380)
> >> >> at
> >> >>
> >> >>
> org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
> >> >> at
> >> >>
> >> >>
> org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
> >> >> at
> >> >>
> >> >>
> org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
> >> >> at
> >> >>
> >> >>
> org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
> >> >> at
> >> >>
> >> >>
> org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
> >> >> at
> >> >>
> >> >>
> org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
> >> >> at
> org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
> >> >> at org.postgresql.Driver.makeConnection(Driver.java:393)
> >> >> at org.postgresql.Driver.connect(Driver.java:267)
> >> >> at java.sql.DriverManager.getConnection(DriverManager.java:571)
> >> >> at java.sql.DriverManager.getConnection(DriverManager.java:233)
> >> >> at
> >> >>
> org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:824)
> >> >> at
> >> >>
> >> >>
> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
> >> >> at
> >> >>
> >> >>
> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
> >> >> ... 7 more
> >> >> Caused by: javax.security.auth.login.LoginException: No LoginModules
> >> >> configured for pgjdbc
> >> >> at javax.security.auth.login.LoginContext.init(LoginContext.java:273)
> >> >> at
> javax.security.auth.login.LoginContext.<init>(LoginContext.java:418)
> >> >> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:42)
> >> >> ... 22 more
> >> >
> >> >
> >
> >
>

Mime
View raw message