sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Lyle <dlyle65...@gmail.com>
Subject Re: Sqoop Import with Postgresql and Kerberos
Date Sat, 22 Nov 2014 18:23:36 GMT
Thanks Abe and Gwen. I was able to get it working- here's what I did:

1) Create pgjdbc.conf with the following containint the following:
    pgjdbc {
         com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true;
    };

2) Put the file in a common location on all the nodemanager nodes on
my cluster. I chose /home/dlyle
3) export HADOOP_OPTS="-Djava.security.auth.login.config=/home/dlyle/pgjdbc.conf"
4) add -Djava.security.auth.login.config=/home/dlyle/pgjdbc.conf to
mapreduce.admin.map.child.java.opts in mapred-site.xml

That did the trick.

Thanks again for the pointers.

-D...


On Fri, Nov 21, 2014 at 4:21 AM, Gwen Shapira <gshapira@cloudera.com> wrote:
> I'd also check /etc/krb5.keytab, you may be running into this issue:
> https://bugzilla.redhat.com/show_bug.cgi?id=748528
>
> (It also happens on Ubuntu)
>
> On Fri, Nov 21, 2014 at 12:54 AM, Abraham Elmahrek <abe@cloudera.com> wrote:
>>
>> Two things I think might be going on:
>>
>> kerberosServerName may need to be just the short name "postgre". Not sure.
>> jaasApplicationName should be set to "Krb5LoginModule". As per
>> https://docs.oracle.com/javase/7/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5LoginModule.html
>> and http://jdbc.postgresql.org/documentation/84/connect.html.
>>
>> -Abe
>>
>> On Thu, Nov 20, 2014 at 5:10 PM, David Lyle <dlyle65535@gmail.com> wrote:
>>>
>>> Thanks for hanging with me. Still same result. Postgres says:
>>>
>>> FATAL: GSSAPI authentication failed for user "dlyle"
>>>
>>> Here's the current state of the world:
>>>
>>> klist -f
>>> Ticket cache: FILE:/tmp/krb5cc_115000008
>>> Default principal: dlyle@LYLECO.COM
>>>
>>> Valid starting     Expires            Service principal
>>> 11/20/14 20:07:20  11/21/14 20:07:19  krbtgt/LYLECO.COM@LYLECO.COM
>>> renew until 11/25/14 20:07:09, Flags: FPRIAT
>>>
>>> [dlyle@hwx1 ~]$ kinit -R
>>> [dlyle@hwx1 ~]$ sqoop list-databases --connect
>>>
>>> "jdbc:postgresql://postgres.lyleco.com:5432?kerberosServerName=postgres/postgres.lylecom.com@LYLECO.COM"
>>> --verbose
>>> kinit: Generic preauthentication failure while getting initial
>>> credentials
>>> kinit: Generic preauthentication failure while getting initial
>>> credentials
>>> 14/11/20 20:07:59 INFO sqoop.Sqoop: Running Sqoop version:
>>> 1.4.4.2.1.7.0-784
>>> 14/11/20 20:07:59 DEBUG tool.BaseSqoopTool: Enabled debug logging.
>>> 14/11/20 20:07:59 DEBUG sqoop.ConnFactory: Loaded manager factory:
>>> com.cloudera.sqoop.manager.DefaultManagerFactory
>>> 14/11/20 20:07:59 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
>>> com.cloudera.sqoop.manager.DefaultManagerFactory
>>> 14/11/20 20:07:59 DEBUG manager.DefaultManagerFactory: Trying with
>>> scheme: jdbc:postgresql:
>>> 14/11/20 20:07:59 INFO manager.SqlManager: Using default fetchSize of
>>> 1000
>>> 14/11/20 20:07:59 DEBUG sqoop.ConnFactory: Instantiated ConnManager
>>> org.apache.sqoop.manager.PostgresqlManager@5cb42b
>>> 14/11/20 20:07:59 DEBUG manager.SqlManager: No connection paramenters
>>> specified. Using regular API for making connection.
>>> 14/11/20 20:07:59 ERROR manager.CatalogQueryManager: Failed to list
>>> databases
>>> org.postgresql.util.PSQLException: GSS Authentication failed
>>> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:49)
>>> at
>>> org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:380)
>>> at
>>> org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
>>> at
>>> org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
>>> at
>>> org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
>>> at
>>> org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
>>> at
>>> org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
>>> at
>>> org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
>>> at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
>>> at org.postgresql.Driver.makeConnection(Driver.java:393)
>>> at org.postgresql.Driver.connect(Driver.java:267)
>>> at java.sql.DriverManager.getConnection(DriverManager.java:571)
>>> at java.sql.DriverManager.getConnection(DriverManager.java:233)
>>> at
>>> org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:824)
>>> at
>>> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
>>> at
>>> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
>>> at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
>>> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
>>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
>>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
>>> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
>>> Caused by: javax.security.auth.login.LoginException: No LoginModules
>>> configured for pgjdbc
>>> at javax.security.auth.login.LoginContext.init(LoginContext.java:273)
>>> at javax.security.auth.login.LoginContext.<init>(LoginContext.java:418)
>>> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:42)
>>> ... 22 more
>>> 14/11/20 20:07:59 ERROR sqoop.Sqoop: Got exception running Sqoop:
>>> java.lang.RuntimeException: org.postgresql.util.PSQLException: GSS
>>> Authentication failed
>>> java.lang.RuntimeException: org.postgresql.util.PSQLException: GSS
>>> Authentication failed
>>> at
>>> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:73)
>>> at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
>>> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
>>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
>>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
>>> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
>>> Caused by: org.postgresql.util.PSQLException: GSS Authentication failed
>>> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:49)
>>> at
>>> org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:380)
>>> at
>>> org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
>>> at
>>> org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
>>> at
>>> org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
>>> at
>>> org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
>>> at
>>> org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
>>> at
>>> org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
>>> at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
>>> at org.postgresql.Driver.makeConnection(Driver.java:393)
>>> at org.postgresql.Driver.connect(Driver.java:267)
>>> at java.sql.DriverManager.getConnection(DriverManager.java:571)
>>> at java.sql.DriverManager.getConnection(DriverManager.java:233)
>>> at
>>> org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:824)
>>> at
>>> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
>>> at
>>> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
>>> ... 7 more
>>> Caused by: javax.security.auth.login.LoginException: No LoginModules
>>> configured for pgjdbc
>>> at javax.security.auth.login.LoginContext.init(LoginContext.java:273)
>>> at javax.security.auth.login.LoginContext.<init>(LoginContext.java:418)
>>> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:42)
>>> ... 22 more
>>>
>>> On Thu, Nov 20, 2014 at 7:24 PM, Abraham Elmahrek <abe@cloudera.com>
>>> wrote:
>>> > Hey there,
>>> >
>>> > My mistake. Apparently the it's a bit more like an HTTP string:
>>> >
>>> > jdbc:postgresql://postgres.lyleco.com:5432?kerberosServerName=postgres/postgres.lyleco.com@LYLECO.COM.
>>> > Also, are the JDBC jars in /var/lib/sqoop? Or are they part of Sqoop's
>>> > class
>>> > path somehow?
>>> >
>>> > The "Generic preauthentication" errors are discomforting. Are your
>>> > credentials renewable? I believe hadoop-auth starts a renewer thread
>>> > that
>>> > runs "kinit -R". That could be causing that error.
>>> >
>>> > -Abe
>>> >
>>> > On Thu, Nov 20, 2014 at 3:56 PM, David Lyle <dlyle65535@gmail.com>
>>> > wrote:
>>> >>
>>> >> Hi Abe,
>>> >>
>>> >> Thanks for the quick reply. Currently, the postgres instance in all
>>> >> that is kerberized. When I add the kerberosServerName I get:
>>> >>
>>> >> java.sql.SQLException: No suitable driver found for
>>> >>
>>> >>
>>> >> jdbc:postgresql://postgres.lyleco.com:5432;kerberosServerName=postgres/postgres.lyleco.com@LYLECO.COM
>>> >>
>>> >> or
>>> >>
>>> >> java.sql.SQLException: No suitable driver found for
>>> >>
>>> >>
>>> >> jdbc:postgresql://postgres.lyleco.com:5432&kerberosServerName=postgres/postgres.lyleco.com@LYLECO.COM
>>> >>
>>> >> (I tried both). I'm also troubled by the repeated kinit: Generic
>>> >> preauthentication failure while getting initial credentials at the
>>> >> very top. I don't understand how sqoop is delegating, but the kinit
is
>>> >> surprising.
>>> >>
>>> >> -D...
>>> >>
>>> >>
>>> >> On Thu, Nov 20, 2014 at 6:15 PM, Abraham Elmahrek <abe@cloudera.com>
>>> >> wrote:
>>> >> > This is a long shot... but I thought I'd say something.
>>> >> >
>>> >> > If you have Hadoop configured for Kerberos, then Sqoop should
>>> >> > automatically
>>> >> > use your kerberos credentials to run a MapReduce job. That MapReduce
>>> >> > job
>>> >> > will run with its own Kerberos principal (normally yarn or mapred).
>>> >> > This
>>> >> > means that Hadoop has access to postgresql as long as it requests
a
>>> >> > session
>>> >> > from the KDC, assuming postgresql and Hadoop are part of the same
>>> >> > realm.
>>> >> > My
>>> >> > understanding is the JDBC driver will handle this for you if you
>>> >> > provide
>>> >> > the
>>> >> > postgresql service name via "kerberosServerName" in the JDBC
>>> >> > connection
>>> >> > string:
>>> >> >
>>> >> >
>>> >> > "jdbc:postgresql://postgres.lyleco.com:5432;kerberosServerName=postgres/<host>@<REALM>".
>>> >> >
>>> >> > Ref: http://jdbc.postgresql.org/documentation/84/connect.html
>>> >> >
>>> >> > -Abe
>>> >> >
>>> >> > On Thu, Nov 20, 2014 at 2:53 PM, David Lyle <dlyle65535@gmail.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> Hi,
>>> >> >>
>>> >> >> I'm trying to use Sqoop to connect to a kerberized Postgres
>>> >> >> instance.
>>> >> >> At this time, I can connect from psql, but my sqoop session
gives
>>> >> >> me
>>> >> >> the log below. Any idea where I'm going wrong?
>>> >> >>
>>> >> >> Thanks!
>>> >> >>
>>> >> >> -David...
>>> >> >>
>>> >> >>
>>> >> >> sqoop list-databases  -Djavax.security.auth.debug=true --connect
>>> >> >> "jdbc:postgresql://postgres.lyleco.com:5432" -verbose
>>> >> >>
>>> >> >> kinit: Generic preauthentication failure while getting initial
>>> >> >> credentials
>>> >> >> kinit: Generic preauthentication failure while getting initial
>>> >> >> credentials
>>> >> >>
>>> >> >> 14/11/20 17:26:37 INFO sqoop.Sqoop: Running Sqoop version:
>>> >> >> 1.4.4.2.1.7.0-784
>>> >> >> 14/11/20 17:26:37 DEBUG tool.BaseSqoopTool: Enabled debug logging.
>>> >> >> 14/11/20 17:26:37 DEBUG sqoop.ConnFactory: Loaded manager factory:
>>> >> >> com.cloudera.sqoop.manager.DefaultManagerFactory
>>> >> >> 14/11/20 17:26:37 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
>>> >> >> com.cloudera.sqoop.manager.DefaultManagerFactory
>>> >> >> 14/11/20 17:26:37 DEBUG manager.DefaultManagerFactory: Trying
with
>>> >> >> scheme: jdbc:postgresql:
>>> >> >> 14/11/20 17:26:37 INFO manager.SqlManager: Using default fetchSize
>>> >> >> of
>>> >> >> 1000
>>> >> >> 14/11/20 17:26:37 DEBUG sqoop.ConnFactory: Instantiated ConnManager
>>> >> >> org.apache.sqoop.manager.PostgresqlManager@1663108
>>> >> >> 14/11/20 17:26:37 DEBUG manager.SqlManager: No connection
>>> >> >> paramenters
>>> >> >> specified. Using regular API for making connection.
>>> >> >> 14/11/20 17:26:37 ERROR manager.CatalogQueryManager: Failed
to list
>>> >> >> databases
>>> >> >> org.postgresql.util.PSQLException: GSS Authentication failed
>>> >> >> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:49)
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:380)
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
>>> >> >> at
>>> >> >> org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
>>> >> >> at org.postgresql.Driver.makeConnection(Driver.java:393)
>>> >> >> at org.postgresql.Driver.connect(Driver.java:267)
>>> >> >> at java.sql.DriverManager.getConnection(DriverManager.java:571)
>>> >> >> at java.sql.DriverManager.getConnection(DriverManager.java:233)
>>> >> >> at
>>> >> >>
>>> >> >> org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:824)
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
>>> >> >> at
>>> >> >>
>>> >> >> org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
>>> >> >> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
>>> >> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>> >> >> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
>>> >> >> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
>>> >> >> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
>>> >> >> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
>>> >> >> Caused by: javax.security.auth.login.LoginException: No
>>> >> >> LoginModules
>>> >> >> configured for pgjdbc
>>> >> >> at
>>> >> >> javax.security.auth.login.LoginContext.init(LoginContext.java:273)
>>> >> >> at
>>> >> >> javax.security.auth.login.LoginContext.<init>(LoginContext.java:418)
>>> >> >> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:42)
>>> >> >> ... 22 more
>>> >> >> 14/11/20 17:26:37 ERROR sqoop.Sqoop: Got exception running
Sqoop:
>>> >> >> java.lang.RuntimeException: org.postgresql.util.PSQLException:
GSS
>>> >> >> Authentication failed
>>> >> >> java.lang.RuntimeException: org.postgresql.util.PSQLException:
GSS
>>> >> >> Authentication failed
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:73)
>>> >> >> at
>>> >> >>
>>> >> >> org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
>>> >> >> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
>>> >> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>> >> >> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
>>> >> >> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
>>> >> >> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
>>> >> >> at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
>>> >> >> Caused by: org.postgresql.util.PSQLException: GSS Authentication
>>> >> >> failed
>>> >> >> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:49)
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:380)
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
>>> >> >> at
>>> >> >> org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
>>> >> >> at org.postgresql.Driver.makeConnection(Driver.java:393)
>>> >> >> at org.postgresql.Driver.connect(Driver.java:267)
>>> >> >> at java.sql.DriverManager.getConnection(DriverManager.java:571)
>>> >> >> at java.sql.DriverManager.getConnection(DriverManager.java:233)
>>> >> >> at
>>> >> >>
>>> >> >> org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:824)
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
>>> >> >> at
>>> >> >>
>>> >> >>
>>> >> >> org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
>>> >> >> ... 7 more
>>> >> >> Caused by: javax.security.auth.login.LoginException: No
>>> >> >> LoginModules
>>> >> >> configured for pgjdbc
>>> >> >> at
>>> >> >> javax.security.auth.login.LoginContext.init(LoginContext.java:273)
>>> >> >> at
>>> >> >> javax.security.auth.login.LoginContext.<init>(LoginContext.java:418)
>>> >> >> at org.postgresql.gss.MakeGSS.authenticate(MakeGSS.java:42)
>>> >> >> ... 22 more
>>> >> >
>>> >> >
>>> >
>>> >
>>
>>
>

Mime
View raw message