Could it be a requirement of the jdbc driver, as this doesn't work  without using the integratedSecurity and authenticationScheme options.

[USER@FQDN~]$ klist -f
Ticket cache: FILE:/tmp/krb5cc_cdc478262072_7Vemq9
Default principal: USER@DOMAIN

Valid starting       Expires              Service principal
09/08/2016 16:51:36  09/09/2016 02:51:36  krbtgt/DOMAIN@DOMAIN
        renew until 09/15/2016 16:51:36, Flags: FRIA
[USER@FQDN~]$ sqoop list-databases --connect "jdbc:sqlserver://FQDN" --username 'DOMAIN\USER'
Warning: /usr/hdp/2.4.2.0-258/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/hdp/2.4.2.0-258/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/09/08 16:54:56 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.2.0-258
16/09/08 16:54:56 INFO manager.SqlManager: Using default fetchSize of 1000
16/09/08 16:54:56 ERROR manager.CatalogQueryManager: Failed to list databases
com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user 'DOMAIN\USER'. ClientConnectionId:eb2fb32a-60be-4e92-9eaa-3fcf79fe95a7
        at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:217)
        at com.microsoft.sqlserver.jdbc.TDSTokenHandler.onEOF(tdsparser.java:279)
        at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:99)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4343)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3159)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.access$100(SQLServerConnection.java:42)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3122)
        at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1980)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1627)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1458)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:772)
        at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1168)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:885)
        at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
        at org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
        at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
16/09/08 16:54:56 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user 'DOMAIN\USER'. ClientConnectionId:eb2fb32a-60be-4e92-9eaa-3fcf79fe95a7
java.lang.RuntimeException: com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user 'DOMAIN\USER'. ClientConnectionId:eb2fb32a-60be-4e92-9eaa-3fcf79fe95a7
        at org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:73)
        at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user 'DOMAIN\USER'. ClientConnectionId:eb2fb32a-60be-4e92-9eaa-3fcf79fe95a7
        at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:217)
        at com.microsoft.sqlserver.jdbc.TDSTokenHandler.onEOF(tdsparser.java:279)
        at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:99)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4343)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3159)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.access$100(SQLServerConnection.java:42)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3122)
        at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1980)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1627)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1458)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:772)
        at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1168)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:885)
        at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
        at org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
        ... 7 more
[u190491a@mlwhdp1g002 ~]$ sqoop list-databases --connect "jdbc:sqlserver://FQDN;integratedSecurity=true;authenticationScheme=JavaKerberos" --username 'DOMAIN\USER'
Warning: /usr/hdp/2.4.2.0-258/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/hdp/2.4.2.0-258/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/09/08 16:55:05 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.2.0-258
16/09/08 16:55:05 INFO manager.SqlManager: Using default fetchSize of 1000
...






Jonathan Kaufman | DevOps Engineer | Foot Locker Inc.
         (414-357-4062) | jkaufman@footlocker.com




From:        billy iftissen <billy.iftissen@gmail.com>
To:        user@sqoop.apache.org
Date:        09/08/2016 05:11 AM
Subject:        Re: Help using Scoop 1.4.6 with Kerberos Authentication on Import with MapReduce.




Hello,
Sqoop 1.4.6 support kerberos ==>  it mean your mapreduce process is lanched by a principal that is in realm of KDC (entry point) , verify ticket with klist
so you dont need to spécify options integratedSecurity=true;authenticationScheme=JavaKerberos;
 
hope this helps you

sqoop import --connect "jdbc:sqlserver://FQDN;database=database_name" --username 'Domain\user' --query 'select * from table where $CONDITIONS' --split-by Abc_Number --target-dir /user/user/SQL

2016-09-07 17:47 GMT+02:00 Jonathan Kaufman <jkaufman@footlocker.com>:
Hello all,

I apologize in advance if this seems like a silly question, however I've not been able to determine if Scoop supports  Kerberos delegation on import, down to the MapReduce process.


I have a user who has a valid Kerberos ticket that supports forwarding, additionally they are working on a completely kerberized Hadoop cluster and all hosts in the cluster are trusted for kerberos delegation.


We have been able to use Scoop with Kerberos Authentication when connecting to a Microsoft SQL Server using the following command:


sqoop

list
-databases --connect
"jdbc:sqlserver://
hostname.domain.com;integratedSecurity=true;authenticationScheme=JavaKerberos"
--
username 'DOMAIN\USER'

So we tried to do an import using Kerberos, but it has failed with an error that indicates that MapReduce does not have a kerberos ticket.


Here is the command used on the Sqoop import.


sqoop import --connect "jdbc:sqlserver://FQDN;integratedSecurity=true;authenticationScheme=JavaKerberos;database=database_name" --username 'Domain\user' --query 'select * from table where $CONDITIONS' --split-by Abc_Number --target-dir /user/user/Sql


Here is the error:

INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:NNHA, Ident: (HDFS_DELEGATION_TOKEN token 2496 for ) INFO impl.YarnClientImpl: Submitted application INFO mapreduce.Job: The url to track the job: INFO mapreduce.Job: Running job: job_1 INFO mapreduce.Job: Job job_1 running in uber mode : false INFO mapreduce.Job: map 0% reduce 0% INFO mapreduce.Job: Task Id : attempt_1470416754637_0238_m_000003_0, Status : FAILED Error: java.lang.RuntimeException: java.lang.RuntimeException: com.microsoft.sqlserver.jdbc.SQLServerException: Integrated authentication failed. ClientConnectionId:366c38c7-c at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: java.lang.RuntimeException: com.microsoft.sqlserver.jdbc.SQLServerException: Integrated authentication failed. ClientConnectionId:366c38c7-c at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220) at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165) ... 9 more Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Integrated authentication failed. ClientConnectionId:366c38c7-c at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2399) at com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:176) at com.microsoft.sqlserver.jdbc.KerbAuthentication.GenerateClientContext(KerbAuthentication.java:296) at com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4084) at com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3159) at com.microsoft.sqlserver.jdbc.SQLServerConnection.access$100(SQLServerConnection.java:42) at com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3122) at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505) at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444) at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1980) at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1627) at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1458) at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:772) at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1168) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:247) at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302) at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213) ... 10 more Caused by: java.security.PrivilegedActionException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at com.microsoft.sqlserver.jdbc.KerbAuthentication.getClientCredential(KerbAuthentication.java:199) at com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:150) ... 26 more Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) at sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193) at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427) at sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:62) at sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154) at com.microsoft.sqlserver.jdbc.KerbAuthentication$1.run(KerbAuthentication.java:189) at com.microsoft.sqlserver.jdbc.KerbAuthentication$1.run(KerbAuthentication.java:187) ... 30 more



Does anyone know if Scoop support forwarding the Kerberos ticket? and if so, what do we need to configure to have it do so? Any comments on how this should work, or websites discussing how you got this working in your environment?


There is a Hortonworks Community post on this as well:  
https://community.hortonworks.com/questions/52137/sqlserver-sqoop-import-integrated-security-kerbero.html

I would like to verify each step of the process can support Kerberos, and then verify the configurations to determine if there is any configuration set that might prevent this from working.


Any help would be appreciated.


thanks.


Jonathan Kaufman | DevOps Engineer | Foot Locker Inc.
         (414-357-4062) |
jkaufman@footlocker.com
Visit us on-line at footlocker.com.



The information in this e-mail, and any attachment therein, is confidential and for use by the addressee only. If you are not the intended recipient, please return the e-mail to the sender and delete it from your computer. Although the Company attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses.




--

Rejoignez moi sur le réseau professionnel viadeo à l'adresse:
http://www.viadeo.com/fr/profile/matouk.iftissen1