sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Kaufman <jkauf...@footlocker.com>
Subject Re: Help using Scoop 1.4.6 with Kerberos Authentication on Import with MapReduce.
Date Thu, 08 Sep 2016 20:59:42 GMT
Could it be a requirement of the jdbc driver, as this doesn't work without 
using the integratedSecurity and authenticationScheme options.

[USER@FQDN~]$ klist -f
Ticket cache: FILE:/tmp/krb5cc_cdc478262072_7Vemq9
Default principal: USER@DOMAIN

Valid starting       Expires              Service principal
09/08/2016 16:51:36  09/09/2016 02:51:36  krbtgt/DOMAIN@DOMAIN
        renew until 09/15/2016 16:51:36, Flags: FRIA
[USER@FQDN~]$ sqoop list-databases --connect "jdbc:sqlserver://FQDN" 
--username 'DOMAIN\USER'
Warning: /usr/hdp/2.4.2.0-258/hbase does not exist! HBase imports will 
fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/hdp/2.4.2.0-258/accumulo does not exist! Accumulo imports 
will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/09/08 16:54:56 INFO sqoop.Sqoop: Running Sqoop version: 
1.4.6.2.4.2.0-258
16/09/08 16:54:56 INFO manager.SqlManager: Using default fetchSize of 1000
16/09/08 16:54:56 ERROR manager.CatalogQueryManager: Failed to list 
databases
com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user 
'DOMAIN\USER'. ClientConnectionId:eb2fb32a-60be-4e92-9eaa-3fcf79fe95a7
        at 
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:217)
        at 
com.microsoft.sqlserver.jdbc.TDSTokenHandler.onEOF(tdsparser.java:279)
        at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:99)
        at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4343)
        at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3159)
        at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.access$100(SQLServerConnection.java:42)
        at 
com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3122)
        at 
com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505)
        at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)
        at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1980)
        at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1627)
        at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1458)
        at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:772)
        at 
com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1168)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at 
org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:885)
        at 
org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
        at 
org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
        at 
org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
16/09/08 16:54:56 ERROR sqoop.Sqoop: Got exception running Sqoop: 
java.lang.RuntimeException: 
com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user 
'DOMAIN\USER'. ClientConnectionId:eb2fb32a-60be-4e92-9eaa-3fcf79fe95a7
java.lang.RuntimeException: 
com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user 
'DOMAIN\USER'. ClientConnectionId:eb2fb32a-60be-4e92-9eaa-3fcf79fe95a7
        at 
org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:73)
        at 
org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Login failed 
for user 'DOMAIN\USER'. 
ClientConnectionId:eb2fb32a-60be-4e92-9eaa-3fcf79fe95a7
        at 
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:217)
        at 
com.microsoft.sqlserver.jdbc.TDSTokenHandler.onEOF(tdsparser.java:279)
        at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:99)
        at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4343)
        at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3159)
        at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.access$100(SQLServerConnection.java:42)
        at 
com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3122)
        at 
com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505)
        at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)
        at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1980)
        at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1627)
        at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1458)
        at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:772)
        at 
com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1168)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at 
org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:885)
        at 
org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
        at 
org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
        ... 7 more
[u190491a@mlwhdp1g002 ~]$ sqoop list-databases --connect 
"jdbc:sqlserver://FQDN;integratedSecurity=true;authenticationScheme=JavaKerberos" 
--username 'DOMAIN\USER'
Warning: /usr/hdp/2.4.2.0-258/hbase does not exist! HBase imports will 
fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/hdp/2.4.2.0-258/accumulo does not exist! Accumulo imports 
will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/09/08 16:55:05 INFO sqoop.Sqoop: Running Sqoop version: 
1.4.6.2.4.2.0-258
16/09/08 16:55:05 INFO manager.SqlManager: Using default fetchSize of 1000
...






Jonathan Kaufman | DevOps Engineer | Foot Locker Inc.
          (414-357-4062) | jkaufman@footlocker.com



From:   billy iftissen <billy.iftissen@gmail.com>
To:     user@sqoop.apache.org
Date:   09/08/2016 05:11 AM
Subject:        Re: Help using Scoop 1.4.6 with Kerberos Authentication on 
Import with MapReduce.



Hello,
Sqoop 1.4.6 support kerberos ==>  it mean your mapreduce process is 
lanched by a principal that is in realm of KDC (entry point) , verify 
ticket with klist 
so you dont need to spécify options 
integratedSecurity=true;authenticationScheme=JavaKerberos;
 
hope this helps you

sqoop import --connect "jdbc:sqlserver://FQDN;database=database_name" 
--username 'Domain\user' --query 'select * from table where $CONDITIONS' 
--split-by Abc_Number --target-dir /user/user/SQL 

2016-09-07 17:47 GMT+02:00 Jonathan Kaufman <jkaufman@footlocker.com>:
Hello all, 

I apologize in advance if this seems like a silly question, however I've 
not been able to determine if Scoop supports  Kerberos delegation on 
import, down to the MapReduce process. 

I have a user who has a valid Kerberos ticket that supports forwarding, 
additionally they are working on a completely kerberized Hadoop cluster 
and all hosts in the cluster are trusted for kerberos delegation. 

We have been able to use Scoop with Kerberos Authentication when 
connecting to a Microsoft SQL Server using the following command: 

sqoop 
list-databases --connect 
"jdbc:sqlserver://hostname.domain.com
;integratedSecurity=true;authenticationScheme=JavaKerberos" 
--username 'DOMAIN\USER' 

So we tried to do an import using Kerberos, but it has failed with an 
error that indicates that MapReduce does not have a kerberos ticket. 

Here is the command used on the Sqoop import. 

sqoop import --connect 
"jdbc:sqlserver://FQDN;integratedSecurity=true;authenticationScheme=JavaKerberos;database=database_name"

--username 'Domain\user' --query 'select * from table where $CONDITIONS' 
--split-by Abc_Number --target-dir /user/user/Sql 

Here is the error: 
INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: 
ha-hdfs:NNHA, Ident: (HDFS_DELEGATION_TOKEN token 2496 for ) INFO 
impl.YarnClientImpl: Submitted application INFO mapreduce.Job: The url to 
track the job: INFO mapreduce.Job: Running job: job_1 INFO mapreduce.Job: 
Job job_1 running in uber mode : false INFO mapreduce.Job: map 0% reduce 
0% INFO mapreduce.Job: Task Id : attempt_1470416754637_0238_m_000003_0, 
Status : FAILED Error: java.lang.RuntimeException: 
java.lang.RuntimeException: 
com.microsoft.sqlserver.jdbc.SQLServerException: Integrated authentication 
failed. ClientConnectionId:366c38c7-c at 
org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167) 
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76) 
at 
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) 
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at 
org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at 
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Subject.java:422) at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) 
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: 
java.lang.RuntimeException: 
com.microsoft.sqlserver.jdbc.SQLServerException: Integrated authentication 
failed. ClientConnectionId:366c38c7-c at 
org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220) 
at 
org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165) 
... 9 more Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: 
Integrated authentication failed. ClientConnectionId:366c38c7-c at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2399)

at 
com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:176) 
at 
com.microsoft.sqlserver.jdbc.KerbAuthentication.GenerateClientContext(KerbAuthentication.java:296)

at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4084)

at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3159) 
at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.access$100(SQLServerConnection.java:42) 
at 
com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3122)

at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505) at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)

at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1980)

at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1627) 
at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1458)

at 
com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:772) 
at 
com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1168) 
at java.sql.DriverManager.getConnection(DriverManager.java:664) at 
java.sql.DriverManager.getConnection(DriverManager.java:247) at 
org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302) 
at 
org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213) 
... 10 more Caused by: java.security.PrivilegedActionException: 
GSSException: No valid credentials provided (Mechanism level: Failed to 
find any Kerberos tgt) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Subject.java:422) at 
com.microsoft.sqlserver.jdbc.KerbAuthentication.getClientCredential(KerbAuthentication.java:199)

at 
com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:150) 
... 26 more Caused by: GSSException: No valid credentials provided 
(Mechanism level: Failed to find any Kerberos tgt) at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) 
at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) 
at 
sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193) 
at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427) at 
sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:62) at 
sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154) 
at 
com.microsoft.sqlserver.jdbc.KerbAuthentication$1.run(KerbAuthentication.java:189) 
at 
com.microsoft.sqlserver.jdbc.KerbAuthentication$1.run(KerbAuthentication.java:187) 
... 30 more 


Does anyone know if Scoop support forwarding the Kerberos ticket? and if 
so, what do we need to configure to have it do so? Any comments on how 
this should work, or websites discussing how you got this working in your 
environment? 

There is a Hortonworks Community post on this as well:  
https://community.hortonworks.com/questions/52137/sqlserver-sqoop-import-integrated-security-kerbero.html



I would like to verify each step of the process can support Kerberos, and 
then verify the configurations to determine if there is any configuration 
set that might prevent this from working. 

Any help would be appreciated. 

thanks. 

Jonathan Kaufman | DevOps Engineer | Foot Locker Inc.
         (414-357-4062) | jkaufman@footlocker.com

Visit us on-line at footlocker.com. 


The information in this e-mail, and any attachment therein, is 
confidential and for use by the addressee only. If you are not the 
intended recipient, please return the e-mail to the sender and delete it 
from your computer. Although the Company attempts to sweep e-mail and 
attachments for viruses, it does not guarantee that either are virus-free 
and accepts no liability for any damage sustained as a result of viruses.



-- 
Rejoignez moi sur le réseau professionnel viadeo à l'adresse:
http://www.viadeo.com/fr/profile/matouk.iftissen1


Mime
View raw message