sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From billy iftissen <billy.iftis...@gmail.com>
Subject Re: Help using Scoop 1.4.6 with Kerberos Authentication on Import with MapReduce.
Date Thu, 08 Sep 2016 10:10:59 GMT
Hello,
Sqoop 1.4.6 support kerberos ==>  it mean your mapreduce process is lanched
by a *principal *that is in realm of KDC (entry point) , verify ticket with
klist
so you dont need to spécify options integratedSecurity=true;
authenticationScheme=JavaKerberos;

hope this helps you

sqoop import --connect "jdbc:sqlserver://FQDN;database=database_name"
--username 'Domain\user' --query 'select * from table where $CONDITIONS'
--split-by Abc_Number --target-dir /user/user/SQL

2016-09-07 17:47 GMT+02:00 Jonathan Kaufman <jkaufman@footlocker.com>:

> Hello all,
>
> I apologize in advance if this seems like a silly question, however I've
> not been able to determine if Scoop supports  Kerberos delegation on
> import, down to the MapReduce process.
>
> I have a user who has a valid Kerberos ticket that supports forwarding,
> additionally they are working on a completely kerberized Hadoop cluster and
> all hosts in the cluster are trusted for kerberos delegation.
>
> We have been able to use Scoop with Kerberos Authentication when
> connecting to a Microsoft SQL Server using the following command:
>
> sqoop
> list-databases --connect
> "jdbc:sqlserver://hostname.domain.com;integratedSecurity=
> true;authenticationScheme=JavaKerberos"
> --username 'DOMAIN\USER'
>
> So we tried to do an import using Kerberos, but it has failed with an
> error that indicates that MapReduce does not have a kerberos ticket.
>
> Here is the command used on the Sqoop import.
>
> sqoop import --connect "jdbc:sqlserver://FQDN;integratedSecurity=true;
> authenticationScheme=JavaKerberos;database=database_name" --username
> 'Domain\user' --query 'select * from table where $CONDITIONS' --split-by
> Abc_Number --target-dir /user/user/Sql
>
> Here is the error:
> INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service:
> ha-hdfs:NNHA, Ident: (HDFS_DELEGATION_TOKEN token 2496 for ) INFO
> impl.YarnClientImpl: Submitted application INFO mapreduce.Job: The url to
> track the job: INFO mapreduce.Job: Running job: job_1 INFO mapreduce.Job:
> Job job_1 running in uber mode : false INFO mapreduce.Job: map 0% reduce 0%
> INFO mapreduce.Job: Task Id : attempt_1470416754637_0238_m_000003_0,
> Status : FAILED Error: java.lang.RuntimeException:
> java.lang.RuntimeException: com.microsoft.sqlserver.jdbc.SQLServerException:
> Integrated authentication failed. ClientConnectionId:366c38c7-c at
> org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1709) at org.apache.hadoop.mapred.
> YarnChild.main(YarnChild.java:162) Caused by: java.lang.RuntimeException:
> com.microsoft.sqlserver.jdbc.SQLServerException: Integrated
> authentication failed. ClientConnectionId:366c38c7-c at
> org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
> at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
> ... 9 more Caused by: com.microsoft.sqlserver.jdbc.SQLServerException:
> Integrated authentication failed. ClientConnectionId:366c38c7-c at
> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2399)
> at com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:176)
> at com.microsoft.sqlserver.jdbc.KerbAuthentication.GenerateClientContext(KerbAuthentication.java:296)
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4084)
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3159)
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.access$
> 100(SQLServerConnection.java:42) at com.microsoft.sqlserver.jdbc.
> SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3122)
> at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505) at
> com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1980)
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1627)
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1458)
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:772)
> at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1168)
> at java.sql.DriverManager.getConnection(DriverManager.java:664) at
> java.sql.DriverManager.getConnection(DriverManager.java:247) at
> org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
> at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
> ... 10 more Caused by: java.security.PrivilegedActionException:
> GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt) at java.security.AccessController.doPrivileged(Native
> Method) at javax.security.auth.Subject.doAs(Subject.java:422) at
> com.microsoft.sqlserver.jdbc.KerbAuthentication.getClientCredential(KerbAuthentication.java:199)
> at com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:150)
> ... 26 more Caused by: GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt) at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
> at sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193)
> at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427) at
> sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:62) at
> sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154)
> at com.microsoft.sqlserver.jdbc.KerbAuthentication$1.run(KerbAuthentication.java:189)
> at com.microsoft.sqlserver.jdbc.KerbAuthentication$1.run(KerbAuthentication.java:187)
> ... 30 more
>
>
> Does anyone know if Scoop support forwarding the Kerberos ticket? and if
> so, what do we need to configure to have it do so? Any comments on how this
> should work, or websites discussing how you got this working in your
> environment?
>
> There is a Hortonworks Community post on this as well:  https://community.
> hortonworks.com/questions/52137/sqlserver-sqoop-import-
> integrated-security-kerbero.html
>
> I would like to verify each step of the process can support Kerberos, and
> then verify the configurations to determine if there is any configuration
> set that might prevent this from working.
>
> Any help would be appreciated.
>
> thanks.
>
> Jonathan Kaufman | DevOps Engineer | Foot Locker Inc.
>          (414-357-4062) | jkaufman@footlocker.com
>
> Visit us on-line at *footlocker.com*
> <http://www.footlocker.com/home/default.cfm?SID=8431>.
>
>
> The information in this e-mail, and any attachment therein, is
> confidential and for use by the addressee only. If you are not the intended
> recipient, please return the e-mail to the sender and delete it from your
> computer. Although the Company attempts to sweep e-mail and attachments for
> viruses, it does not guarantee that either are virus-free and accepts no
> liability for any damage sustained as a result of viruses.




-- 
Rejoignez moi sur le réseau professionnel viadeo à l'adresse:
http://www.viadeo.com/fr/profile/matouk.iftissen1

Mime
View raw message