I was able to isolate this problem to the Sqoop side not picking up correct kerberos credentials. Hbase is picking up the correct kerberos credentials when Hbase put and scan are done in isolation without using Sqoop. 

A direct map-reduce put into HBase uses the following 2 methods - 
HBaseConfiguration.merge(conf, HBaseConfiguration.create(conf)); 
TableMapReduceUtil.initCredentials(job); 

I was looking at how sqoop 1.4.3 does HBase puts to see if it converts sqoop import arguments into map-reduce jobs and uses the above methods somewhere. This is what I found - 
HBasePutProcessor.java  - SqoopRecordProcessor that performs a HBase "put" operation - has a method to get hadoop configuration, but none to merge any kerberos specific configurations specified  in sqoop-site.xml- 

  public Configuration getConf() {
    return this.conf;



HBaseUtil.java   - makes sure hbase jars are present on class path 
PutTransformer.java  - converts jdbc statements in the form of K-V map into hbase put commands and returns a list
ToStringPutTransformer.java - extends the above class 

Does anyone know sqoop internals of how to specify kerberos configurations and get sqoop to read them? 

Cheers,
Suhas.


On Tue, Aug 6, 2013 at 10:31 AM, Suhas Satish <suhas.satish@gmail.com> wrote:
Ataching the logs here at the time of authentication, I do not see any error msges here. 

/var/log/kadmind.log
/var/log/krb5kdc.log

Please let me know if there is any other places I can find other log files 

Cheers,
Suhas.


On Mon, Aug 5, 2013 at 4:48 PM, Abraham Elmahrek <abe@cloudera.com> wrote:
User,

Could you please provide your KDC logs around the time you tried to authenticate?

Note: A kerberos client will negotiate the encryption algorithm it can/will use with the KDC. It may choose AES-256.

-Abe


On Mon, Aug 5, 2013 at 3:55 PM, Suhas Satish <suhas.satish@gmail.com> wrote:
I generated a keytab with the following cmd and it supports multiple encryption types other than aes256 as listed below. 
But I still get the same error from sqoop import tool because the sqoop.keytab is not being read (sqoop being the hbase client in this case). 
 
kadmin:  ktadd -k sqoop.keytab kuser1
Entry for principal kuser1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:sqoop.keytab.
Entry for principal kuser1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:sqoop.keytab.
Entry for principal kuser1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:sqoop.keytab.
Entry for principal kuser1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:sqoop.keytab.
Entry for principal kuser1 with kvno 2, encryption type des-hmac-sha1 added to keytab WRFILE:sqoop.keytab.
Entry for principal kuser1 with kvno 2, encryption type des-cbc-md5 added to keytab WRFILE:sqoop.keytab.

Here are some more debug logs I obtained from kerberos - 

kadmin:  getprinc kuser1
Principal: kuser1@QA.LAB
Expiration date: [never]
Last password change: Mon Aug 05 15:40:30 PDT 2013
Password expiration date: [none]
Maximum ticket life: 1 day 00:00:00
Maximum renewable life: 0 days 00:00:00
Last modified: Mon Aug 05 15:40:30 PDT 2013 (mapr/admin@QA.LAB)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 6
Key: vno 2, aes256-cts-hmac-sha1-96, no salt
Key: vno 2, aes128-cts-hmac-sha1-96, no salt
Key: vno 2, des3-cbc-sha1, no salt
Key: vno 2, arcfour-hmac, no salt
Key: vno 2, des-hmac-sha1, no salt
Key: vno 2, des-cbc-md5, no salt
MKey: vno 1
Attributes:
Policy: [none]

getprinc hbase/qa-node133.qa.lab
Principal: hbase/qa-node133.qa.lab@QA.LAB
Expiration date: [never]
Last password change: Mon Jul 29 19:17:46 PDT 2013
Password expiration date: [none]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Mon Jul 29 19:17:46 PDT 2013 (kuser1/admin@QA.LAB)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 6
Key: vno 2, aes256-cts-hmac-sha1-96, no salt
Key: vno 2, aes128-cts-hmac-sha1-96, no salt
Key: vno 2, des3-cbc-sha1, no salt
Key: vno 2, arcfour-hmac, no salt
Key: vno 2, des-hmac-sha1, no salt
Key: vno 2, des-cbc-md5, no salt
MKey: vno 1
Attributes:
Policy: [none]


Thanks,
Suhas.


On Mon, Aug 5, 2013 at 2:29 PM, Abraham Elmahrek <abe@cloudera.com> wrote:
There should be a password. You should have a keytab associated with that principal, which would allow you to authenticate as that principal. See http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Security-Guide/CDH4-Security-Guide.html for more details on how that works.

A couple of things...
1. You need to make your kerberos credentials renewable. Right now it seems like you cannot renew. See http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Security-Guide/cdh4sg_topic_17.html.
2. AES256 encryption is not inherently supported. Did you install support for AES256?

-Abe


On Mon, Aug 5, 2013 at 1:53 PM, Suhas Satish <suhas.satish@gmail.com> wrote:
klist -e -v 

Ticket cache: FILE:/tmp/krb5cc_0
Default principal: kuser1@QA.LAB

Valid starting     Expires            Service principal
08/05/13 12:34:42  08/05/13 22:34:42  krbtgt/QA.LAB@QA.LAB
renew until 08/05/13 12:34:42, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 

Kerberos 5 version 1.10.3

The principal in hbase-site.xml is 
hbase/qa-node133.qa.lab@QA.LAB

How do I create a credential using kinit matching that in hbase-site.xml?  kinit  hbase/qa-node133.qa.lab   throws an error msg 
kinit: Password incorrect while getting initial credentials
although I know that there is no password for that principal.



Cheers,
Suhas.


On Mon, Aug 5, 2013 at 12:52 PM, Abraham Elmahrek <abe@cloudera.com> wrote:
Hi there,

It seems like your client isn't authenticated in both cases. You seem to be receiving errors from HBase and Sqoop. Sqoop 1.4.3 should simply work if your user is already authenticated. Internally, Sqoop is generating delegation tokens to communicate with HBase. It cannot do that without being properly authenticated first though.

Could you provide the output of the following command:
"klist -e -v"

-Abe


On Mon, Aug 5, 2013 at 12:15 PM, Suhas Satish <suhas.satish@gmail.com> wrote:
I have configured hbase 94.9  with kerberos successfully for authentication and authorization as mentioned in the CDH security docs. I am using sqoop 1.4.3. Is there any configuration required from the sqoop client side for kerberos?

I have the following permissions on hbase tables - 
hbase(main):003:0> grant 'kuser1', 'RWXCA', 'demo'
ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=kuser1, scope=demo, family=, qualifer=, action=ADMIN)


bin/sqoop import --connect jdbc:mysql://10.10.1.10/TestDB    --table t1  --hbase-table  t1  --column-family world


When I try to import into it using sqoop with the above cmd, I get the following error - 


2013-08-05 11:59:33,121 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: 
org.apache.hadoop.hbase.security.AccessDeniedException: Token generation only allowed for Kerberos authenticated clients
at org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:87)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion.java:5576)
at org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocessor(HRegionServer.java:3868)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Server.call(SecureRpcEngine.java:308)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)


Cheers,
Suhas.