sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abraham Elmahrek <...@cloudera.com>
Subject Re: Sqoop Import with Postgresql and Kerberos
Date Thu, 20 Nov 2014 23:15:45 GMT
This is a long shot... but I thought I'd say something.

If you have Hadoop configured for Kerberos, then Sqoop should automatically
use your kerberos credentials to run a MapReduce job. That MapReduce job
will run with its own Kerberos principal (normally yarn or mapred). This
means that Hadoop has access to postgresql as long as it requests a session
from the KDC, assuming postgresql and Hadoop are part of the same realm. My
understanding is the JDBC driver will handle this for you if you provide
the postgresql service name via "kerberosServerName" in the JDBC connection
string: "jdbc:postgresql://postgres.lyleco.com:5432
;kerberosServerName=postgres/<host>@<REALM>".

Ref: http://jdbc.postgresql.org/documentation/84/connect.html

-Abe

On Thu, Nov 20, 2014 at 2:53 PM, David Lyle <dlyle65535@gmail.com> wrote:

> Hi,
>
> I'm trying to use Sqoop to connect to a kerberized Postgres instance.
> At this time, I can connect from psql, but my sqoop session gives me
> the log below. Any idea where I'm going wrong?
>
> Thanks!
>
> -David...
>
>
> sqoop list-databases  -Djavax.security.auth.debug=true --connect
> "jdbc:postgresql://postgres.lyleco.com:5432" -verbose
>
> kinit: Generic preauthentication failure while getting initial credentials
> kinit: Generic preauthentication failure while getting initial credentials
>
> 14/11/20 17:26:37 INFO sqoop.Sqoop: Running Sqoop version:
> 1.4.4.2.1.7.0-784
> 14/11/20 17:26:37 DEBUG tool.BaseSqoopTool: Enabled debug logging.
> 14/11/20 17:26:37 DEBUG sqoop.ConnFactory: Loaded manager factory:
> com.cloudera.sqoop.manager.DefaultManagerFactory
> 14/11/20 17:26:37 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
> com.cloudera.sqoop.manager.DefaultManagerFactory
> 14/11/20 17:26:37 DEBUG manager.DefaultManagerFactory: Trying with
> scheme: jdbc:postgresql:
> 14/11/20 17:26:37 INFO manager.SqlManager: Using default fetchSize of 1000
> 14/11/20 17:26:37 DEBUG sqoop.ConnFactory: Instantiated ConnManager
> org.apache.sqoop.manager.PostgresqlManager@1663108
> 14/11/20 17:26:37 DEBUG manager.SqlManager: No connection paramenters
> specified. Using regular API for making connection.
> 14/11/20 17:26:37 ERROR manager.CatalogQueryManager: Failed to list
> databases
> org.postgresql.util.PSQLException: GSS Authentication failed
> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:49)
> at
> org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:380)
> at
> org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
> at
> org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
> at
> org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
> at
> org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
> at
> org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
> at
> org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
> at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
> at org.postgresql.Driver.makeConnection(Driver.java:393)
> at org.postgresql.Driver.connect(Driver.java:267)
> at java.sql.DriverManager.getConnection(DriverManager.java:571)
> at java.sql.DriverManager.getConnection(DriverManager.java:233)
> at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:824)
> at
> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
> at
> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
> at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
> Caused by: javax.security.auth.login.LoginException: No LoginModules
> configured for pgjdbc
> at javax.security.auth.login.LoginContext.init(LoginContext.java:273)
> at javax.security.auth.login.LoginContext.<init>(LoginContext.java:418)
> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:42)
> ... 22 more
> 14/11/20 17:26:37 ERROR sqoop.Sqoop: Got exception running Sqoop:
> java.lang.RuntimeException: org.postgresql.util.PSQLException: GSS
> Authentication failed
> java.lang.RuntimeException: org.postgresql.util.PSQLException: GSS
> Authentication failed
> at
> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:73)
> at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
> Caused by: org.postgresql.util.PSQLException: GSS Authentication failed
> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:49)
> at
> org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:380)
> at
> org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
> at
> org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
> at
> org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
> at
> org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
> at
> org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
> at
> org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
> at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
> at org.postgresql.Driver.makeConnection(Driver.java:393)
> at org.postgresql.Driver.connect(Driver.java:267)
> at java.sql.DriverManager.getConnection(DriverManager.java:571)
> at java.sql.DriverManager.getConnection(DriverManager.java:233)
> at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:824)
> at
> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
> at
> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
> ... 7 more
> Caused by: javax.security.auth.login.LoginException: No LoginModules
> configured for pgjdbc
> at javax.security.auth.login.LoginContext.init(LoginContext.java:273)
> at javax.security.auth.login.LoginContext.<init>(LoginContext.java:418)
> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:42)
> ... 22 more
>

Mime
View raw message