sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abraham Elmahrek <...@cloudera.com>
Subject Re: sqoop import into secure Hbase with kerberos
Date Tue, 06 Aug 2013 18:23:13 GMT
Sorry, apparently this is an HBase specific token. See here
http://wiki.apache.org/hadoop/Hbase/HBaseTokenAuthentication.


On Tue, Aug 6, 2013 at 11:13 AM, Abraham Elmahrek <abe@cloudera.com> wrote:

> Suhas,
>
> Sqoop 1.4.3 simply fetches the authenticated user from credentials cache
> and fetches a delegation token for HBase. See
> https://issues.apache.org/jira/browse/SQOOP-599 for more information.
>
> -Abe
>
>
> On Tue, Aug 6, 2013 at 11:09 AM, Suhas Satish <suhas.satish@gmail.com>wrote:
>
>> I was able to isolate this problem to the Sqoop side not picking up
>> correct kerberos credentials. Hbase is picking up the correct kerberos
>> credentials when Hbase put and scan are done in isolation without using
>> Sqoop.
>>
>> A direct map-reduce put into HBase uses the following 2 methods -
>> HBaseConfiguration.merge(conf, HBaseConfiguration.create(conf));
>> TableMapReduceUtil.initCredentials(job);
>>
>> I was looking at how sqoop 1.4.3 does HBase puts to see if it converts
>> sqoop import arguments into map-reduce jobs and uses the above methods
>> somewhere. This is what I found -
>> HBasePutProcessor.java  - SqoopRecordProcessor that performs a HBase
>> "put" operation - has a method to get hadoop configuration, but none to
>> merge any kerberos specific configurations specified  in sqoop-site.xml-
>>
>>   public Configuration getConf() {
>>     return this.conf;
>>
>>
>>
>> HBaseUtil.java   - makes sure hbase jars are present on class path
>> PutTransformer.java  - converts jdbc statements in the form of K-V map
>> into hbase put commands and returns a list
>> ToStringPutTransformer.java - extends the above class
>>
>>  Does anyone know sqoop internals of how to specify kerberos
>> configurations and get sqoop to read them?
>>
>> Cheers,
>> Suhas.
>>
>>
>> On Tue, Aug 6, 2013 at 10:31 AM, Suhas Satish <suhas.satish@gmail.com>wrote:
>>
>>> Ataching the logs here at the time of authentication, I do not see any
>>> error msges here.
>>>
>>> /var/log/kadmind.log
>>> /var/log/krb5kdc.log
>>>
>>> Please let me know if there is any other places I can find other log
>>> files
>>>
>>> Cheers,
>>> Suhas.
>>>
>>>
>>> On Mon, Aug 5, 2013 at 4:48 PM, Abraham Elmahrek <abe@cloudera.com>wrote:
>>>
>>>> User,
>>>>
>>>> Could you please provide your KDC logs around the time you tried to
>>>> authenticate?
>>>>
>>>> Note: A kerberos client will negotiate the encryption algorithm it
>>>> can/will use with the KDC. It may choose AES-256.
>>>>
>>>> -Abe
>>>>
>>>>
>>>> On Mon, Aug 5, 2013 at 3:55 PM, Suhas Satish <suhas.satish@gmail.com>wrote:
>>>>
>>>>> I generated a keytab with the following cmd and it supports multiple
>>>>> encryption types other than aes256 as listed below.
>>>>> But I still get the same error from sqoop import tool because the
>>>>> sqoop.keytab is not being read (sqoop being the hbase client in this
case).
>>>>>
>>>>> kadmin:  ktadd -k sqoop.keytab kuser1
>>>>> Entry for principal kuser1 with kvno 2, encryption type
>>>>> aes256-cts-hmac-sha1-96 added to keytab WRFILE:sqoop.keytab.
>>>>> Entry for principal kuser1 with kvno 2, encryption type
>>>>> aes128-cts-hmac-sha1-96 added to keytab WRFILE:sqoop.keytab.
>>>>> Entry for principal kuser1 with kvno 2, encryption type des3-cbc-sha1
>>>>> added to keytab WRFILE:sqoop.keytab.
>>>>> Entry for principal kuser1 with kvno 2, encryption type arcfour-hmac
>>>>> added to keytab WRFILE:sqoop.keytab.
>>>>> Entry for principal kuser1 with kvno 2, encryption type des-hmac-sha1
>>>>> added to keytab WRFILE:sqoop.keytab.
>>>>> Entry for principal kuser1 with kvno 2, encryption type des-cbc-md5
>>>>> added to keytab WRFILE:sqoop.keytab.
>>>>>
>>>>> Here are some more debug logs I obtained from kerberos -
>>>>>
>>>>> *kadmin:  getprinc kuser1*
>>>>> Principal: kuser1@QA.LAB
>>>>> Expiration date: [never]
>>>>> Last password change: Mon Aug 05 15:40:30 PDT 2013
>>>>> Password expiration date: [none]
>>>>> Maximum ticket life: 1 day 00:00:00
>>>>> Maximum renewable life: 0 days 00:00:00
>>>>> Last modified: Mon Aug 05 15:40:30 PDT 2013 (mapr/admin@QA.LAB)
>>>>> Last successful authentication: [never]
>>>>> Last failed authentication: [never]
>>>>> Failed password attempts: 0
>>>>> Number of keys: 6
>>>>> Key: vno 2, aes256-cts-hmac-sha1-96, no salt
>>>>> Key: vno 2, aes128-cts-hmac-sha1-96, no salt
>>>>> Key: vno 2, des3-cbc-sha1, no salt
>>>>> Key: vno 2, arcfour-hmac, no salt
>>>>> Key: vno 2, des-hmac-sha1, no salt
>>>>> Key: vno 2, des-cbc-md5, no salt
>>>>> MKey: vno 1
>>>>> Attributes:
>>>>> Policy: [none]
>>>>>
>>>>> *getprinc hbase/qa-node133.qa.lab*
>>>>> Principal: hbase/qa-node133.qa.lab@QA.LAB
>>>>> Expiration date: [never]
>>>>> Last password change: Mon Jul 29 19:17:46 PDT 2013
>>>>> Password expiration date: [none]
>>>>> Maximum ticket life: 0 days 10:00:00
>>>>> Maximum renewable life: 7 days 00:00:00
>>>>> Last modified: Mon Jul 29 19:17:46 PDT 2013 (kuser1/admin@QA.LAB)
>>>>> Last successful authentication: [never]
>>>>> Last failed authentication: [never]
>>>>> Failed password attempts: 0
>>>>> Number of keys: 6
>>>>> Key: vno 2, aes256-cts-hmac-sha1-96, no salt
>>>>> Key: vno 2, aes128-cts-hmac-sha1-96, no salt
>>>>> Key: vno 2, des3-cbc-sha1, no salt
>>>>> Key: vno 2, arcfour-hmac, no salt
>>>>> Key: vno 2, des-hmac-sha1, no salt
>>>>> Key: vno 2, des-cbc-md5, no salt
>>>>> MKey: vno 1
>>>>> Attributes:
>>>>> Policy: [none]
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Suhas.
>>>>>
>>>>>
>>>>> On Mon, Aug 5, 2013 at 2:29 PM, Abraham Elmahrek <abe@cloudera.com>wrote:
>>>>>
>>>>>> There should be a password. You should have a keytab associated with
>>>>>> that principal, which would allow you to authenticate as that principal.
>>>>>> See
>>>>>> http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Security-Guide/CDH4-Security-Guide.htmlfor
more details on how that works.
>>>>>>
>>>>>> A couple of things...
>>>>>> 1. You need to make your kerberos credentials renewable. Right now
it
>>>>>> seems like you cannot renew. See
>>>>>> http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Security-Guide/cdh4sg_topic_17.html
>>>>>> .
>>>>>> 2. AES256 encryption is not inherently supported. Did you install
>>>>>> support for AES256?
>>>>>>
>>>>>> -Abe
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 5, 2013 at 1:53 PM, Suhas Satish <suhas.satish@gmail.com>wrote:
>>>>>>
>>>>>>> klist -e -v
>>>>>>>
>>>>>>> Ticket cache: FILE:/tmp/krb5cc_0
>>>>>>> Default principal: kuser1@QA.LAB
>>>>>>>
>>>>>>> Valid starting     Expires            Service principal
>>>>>>> 08/05/13 12:34:42  08/05/13 22:34:42  krbtgt/QA.LAB@QA.LAB
>>>>>>> renew until 08/05/13 12:34:42, Etype (skey, tkt):
>>>>>>> aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
>>>>>>>
>>>>>>> Kerberos 5 version 1.10.3
>>>>>>>
>>>>>>> The principal in hbase-site.xml is
>>>>>>> hbase/qa-node133.qa.lab@QA.LAB
>>>>>>>
>>>>>>> How do I create a credential using kinit matching that in
>>>>>>> hbase-site.xml?  kinit  hbase/qa-node133.qa.lab   throws an error
msg
>>>>>>> *kinit: Password incorrect while getting initial credentials*
>>>>>>> *although I know that there is no password for that principal.
*
>>>>>>> *
>>>>>>> *
>>>>>>> *
>>>>>>> *
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Suhas.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Aug 5, 2013 at 12:52 PM, Abraham Elmahrek <abe@cloudera.com>wrote:
>>>>>>>
>>>>>>>> Hi there,
>>>>>>>>
>>>>>>>> It seems like your client isn't authenticated in both cases.
You
>>>>>>>> seem to be receiving errors from HBase and Sqoop. Sqoop 1.4.3
should simply
>>>>>>>> work if your user is already authenticated. Internally, Sqoop
is generating
>>>>>>>> delegation tokens to communicate with HBase. It cannot do
that without
>>>>>>>> being properly authenticated first though.
>>>>>>>>
>>>>>>>> Could you provide the output of the following command:
>>>>>>>> "klist -e -v"
>>>>>>>>
>>>>>>>> -Abe
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Aug 5, 2013 at 12:15 PM, Suhas Satish <
>>>>>>>> suhas.satish@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I have configured hbase 94.9  with kerberos successfully
for
>>>>>>>>> authentication and authorization as mentioned in the
CDH security docs. I
>>>>>>>>> am using sqoop 1.4.3. Is there any configuration required
from the sqoop
>>>>>>>>> client side for kerberos?
>>>>>>>>>
>>>>>>>>> I have the following permissions on hbase tables -
>>>>>>>>> hbase(main):003:0> grant 'kuser1', 'RWXCA', 'demo'
>>>>>>>>> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException:
>>>>>>>>> org.apache.hadoop.hbase.security.AccessDeniedException:
Insufficient
>>>>>>>>> permissions (user=kuser1, scope=demo, family=, qualifer=,
action=ADMIN)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> bin/sqoop import --connect jdbc:mysql://10.10.1.10/TestDB
>>>>>>>>>  --table t1  --hbase-table  t1  --column-family world
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> When I try to import into it using sqoop with the above
cmd, I get
>>>>>>>>> the following error -
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013-08-05 11:59:33,121 ERROR
>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>>>>>> org.apache.hadoop.hbase.security.AccessDeniedException:
Token
>>>>>>>>> generation only allowed for Kerberos authenticated clients
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:87)
>>>>>>>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
>>>>>>>>> at
>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>>>>>  at
>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>  at
>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion.java:5576)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocessor(HRegionServer.java:3868)
>>>>>>>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
>>>>>>>>> at
>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>>>>>  at
>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>  at
>>>>>>>>> org.apache.hadoop.hbase.ipc.SecureRpcEngine$Server.call(SecureRpcEngine.java:308)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Suhas.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message