hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anil Gupta <anilgupt...@gmail.com>
Subject Re: Problem with HBase + Kerberos
Date Mon, 31 Aug 2015 17:06:29 GMT
Yes there is expiration of kerberos session. But, it renews by itself.
As per log, your problem seems to be related to invalid credentials or trouble doing login.


> Couldn't
> setup connection for hbase/host@REALM.WL to hbase/server@REALM.WL

Sent from my iPhone

> On Aug 31, 2015, at 1:43 AM, Loïc Chanel <loic.chanel@telecomnancy.net> wrote:
> 
> Actually, it seems like it is related to Kerberos, as I keep having these
> lines in RegionServers' logs :
> 
> 2015-08-31 10:15:23,370 DEBUG [regionserver60020]
> security.HBaseSaslRpcClient: Creating SASL GSSAPI client. Server's Kerberos
> principal name is hbase/server@REALM.WL
> 2015-08-31 10:15:23,372 DEBUG [regionserver60020] ipc.RpcClient: Exception
> encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> 2015-08-31 10:15:23,372 WARN  [regionserver60020]
> security.UserGroupInformation: Not attempting to re-login since the last
> re-login was attempted less than 600 seconds before.
> 2015-08-31 10:15:27,908 DEBUG [regionserver60020]
> security.HBaseSaslRpcClient: Creating SASL GSSAPI client. Server's Kerberos
> principal name is hbase/server@REALM.WL
> 2015-08-31 10:15:27,910 WARN  [regionserver60020] ipc.RpcClient: 
> 2015-08-31 10:15:27,910 WARN  [regionserver60020]
> regionserver.HRegionServer: error telling master we are up
> com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup
> connection forhbase/host@REALM.WL to hbase/server@REALM.WL
>        at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1739)
>        at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)
>        at
> org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:5402)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2114)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:877)
>        at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Couldn't setup connection for
> hbase/host@REALM.WL to hbase/server@REALM.WL
>        at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at javax.security.auth.Subject.doAs(Unknown Source)
>        at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>        at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
>        at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
>        at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
>        at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
>        at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)
>        at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)
>        ... 5 more
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused
> by GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
>        at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(Unknown
> Source)
>        at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:177)
>        at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)
>        at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)
>        at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)
>        at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at javax.security.auth.Subject.doAs(Unknown Source)
>        at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>        at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)
>        ... 9 more
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
>        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Unknown
> Source)
>        at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Unknown Source)
>        at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Unknown Source)
>        at sun.security.jgss.GSSManagerImpl.getMechanismContext(Unknown
> Source)
>        at sun.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
>        at sun.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
>        ... 19 more
> 2015-08-31 10:15:27,911 WARN  [regionserver60020]
> regionserver.HRegionServer: reportForDuty failed; sleeping and then
> retrying.
> 
> Is there kind of an expiration limit for keytab credentials ?
> Thanks for your help,
> 
> 
> Loïc
> 
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
> 
> 2015-08-27 18:24 GMT+02:00 anil gupta <anilgupta84@gmail.com>:
> 
>> Maybe, this is related to some Ambari setup? Can you also ask on Ambari
>> mailing list.
>> IMO, secure HBase cluster connectivity has been working in HBase for a very
>> long time.
>> 
>> On Thu, Aug 27, 2015 at 12:48 AM, Loïc Chanel <
>> loic.chanel@telecomnancy.net>
>> wrote:
>> 
>>> I did not, but as I Kerberized my cluster with Ambari, it did the
>> mandatory
>>> modifications.
>>> 
>>> Loïc CHANEL
>>> Engineering student at TELECOM Nancy
>>> Trainee at Worldline - Villeurbanne
>>> 
>>> 2015-08-27 1:17 GMT+02:00 Laurent H <laurent.hatier@gmail.com>:
>>> 
>>>> Do you change some stuff in your hbase-site.xml when you've installed
>>>> Kerberos ?
>>>> 
>>>> --
>>>> Laurent HATIER - Consultant Big Data & Business Intelligence chez
>>> CapGemini
>>>> fr.linkedin.com/pub/laurent-hatier/25/36b/a86/
>>>> <http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>
>>>> 
>>>> 2015-08-21 9:44 GMT+02:00 Loïc Chanel <loic.chanel@telecomnancy.net>:
>>>> 
>>>>> Sorry if I didn't mention that, but yeah, I ran kinit before invoking
>>>> hbase
>>>>> shell, and klists command says that my user has a ticket.
>>>>> [root@host /]# klist
>>>>> Ticket cache: FILE:/tmp/krb5cc_0
>>>>> Default principal: testuser@REALM
>>>>> 
>>>>> Valid starting     Expires            Service principal
>>>>> 08/21/15 09:39:33  08/22/15 09:39:33  krbtgt/REALM@REALM
>>>>>        renew until 08/21/15 09:39:33
>>>>> 
>>>>> 
>>>>> Loïc CHANEL
>>>>> Engineering student at TELECOM Nancy
>>>>> Trainee at Worldline - Villeurbanne
>>>>> 
>>>>> 2015-08-21 6:12 GMT+02:00 anil gupta <anilgupta84@gmail.com>:
>>>>> 
>>>>>> Did you run kinit command before invoking "hbase shell"? What does
>>>> klist
>>>>>> command says?
>>>>>> 
>>>>>> On Thu, Aug 20, 2015 at 6:47 AM, Loïc Chanel <
>>>>> loic.chanel@telecomnancy.net
>>>>>> wrote:
>>>>>> 
>>>>>>> By the way, as this may help to find my issue, I just tested
>> typing
>>>>>> *whoami
>>>>>>> *in HBase shell : this returned me exactly what it should :
>>>>>>> testuser@REALM (auth:KERBEROS)
>>>>>>>    groups: nobody, toast
>>>>>>> 
>>>>>>> Loïc CHANEL
>>>>>>> Engineering student at TELECOM Nancy
>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>> 
>>>>>>> 2015-08-20 15:17 GMT+02:00 Loïc Chanel <
>>> loic.chanel@telecomnancy.net
>>>>> :
>>>>>>> 
>>>>>>>> Nothing more with your option :/
>>>>>>>> 
>>>>>>>> Loïc CHANEL
>>>>>>>> Engineering student at TELECOM Nancy
>>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>> 
>>>>>>>> 2015-08-20 15:04 GMT+02:00 Loïc Chanel <
>>>> loic.chanel@telecomnancy.net
>>>>>> :
>>>>>>>> 
>>>>>>>>> I'm using HDP 2.2.4.2, with HBase 0.98.4.2.2.
>>>>>>>>> I have unlimited strength JCE installed.
>>>>>>>>> 
>>>>>>>>> I'll try to have more clues with this option.
>>>>>>>>> 
>>>>>>>>> Loïc CHANEL
>>>>>>>>> Engineering student at TELECOM Nancy
>>>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>>> 
>>>>>>>>> 2015-08-20 14:58 GMT+02:00 Ted Yu <yuzhihong@gmail.com>:
>>>>>>>>> 
>>>>>>>>>> Which hbase / hadoop release are you using ?
>>>>>>>>>> 
>>>>>>>>>> Running with -Dsun.security.krb5.debug=true will
provide more
>>>> clue.
>>>>>>>>>> 
>>>>>>>>>> Do you have unlimited strength JCE installed ?
>>>>>>>>>> 
>>>>>>>>>> Cheers
>>>>>>>>>> 
>>>>>>>>>> On Thu, Aug 20, 2015 at 5:46 AM, Loïc Chanel <
>>>>>>>>>> loic.chanel@telecomnancy.net>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi all,
>>>>>>>>>>> 
>>>>>>>>>>> Since I kerberized my cluster, it seems like
I can't use
>>> HBase
>>>>>>> anymore
>>>>>>>>>> ...
>>>>>>>>>>> For example, executing  create 'toto','titi'
on HBase shell
>>>>> results
>>>>>>> in
>>>>>>>>>> the
>>>>>>>>>>> printing of this line endlessly :
>>>>>>>>>>> WARN  [main] security.UserGroupInformation: Not
attempting
>> to
>>>>>>> re-login
>>>>>>>>>>> since the last re-login was attempted less than
600 seconds
>>>>> before.
>>>>>>>>>>> 
>>>>>>>>>>> And nothing else happens.
>>>>>>>>>>> I tried to restart HDFS and HBase, and to re-generate
>>>> credentials
>>>>>> and
>>>>>>>>>>> keytabs, but nothing changed.
>>>>>>>>>>> As for the logs, they are not very explicits,
as the only
>>> thing
>>>>>> they
>>>>>>>>>> say
>>>>>>>>>>> (and keep saying) is :
>>>>>>>>>>> 
>>>>>>>>>>> 2015-08-20 13:50:12,697 DEBUG
>> [RpcServer.reader=2,port=60000]
>>>>>>>>>>> ipc.RpcServer: Created SASL server with mechanism
= GSSAPI
>>>>>>>>>>> 2015-08-20 13:50:12,698 DEBUG
>> [RpcServer.reader=2,port=60000]
>>>>>>>>>>> ipc.RpcServer: Have read input token of size
650 for
>>> processing
>>>>> by
>>>>>>>>>>> saslServer.evaluateResponse()
>>>>>>>>>>> 2015-08-20 13:50:12,704 DEBUG
>> [RpcServer.reader=2,port=60000]
>>>>>>>>>>> ipc.RpcServer: Will send token of size 108 from
saslServer.
>>>>>>>>>>> 2015-08-20 13:50:12,706 DEBUG
>> [RpcServer.reader=2,port=60000]
>>>>>>>>>>> ipc.RpcServer: Have read input token of size
0 for
>> processing
>>>> by
>>>>>>>>>>> saslServer.evaluateResponse()
>>>>>>>>>>> 2015-08-20 13:50:12,707 DEBUG
>> [RpcServer.reader=2,port=60000]
>>>>>>>>>>> ipc.RpcServer: Will send token of size 32 from
saslServer.
>>>>>>>>>>> 2015-08-20 13:50:12,708 DEBUG
>> [RpcServer.reader=2,port=60000]
>>>>>>>>>>> ipc.RpcServer: RpcServer.listener,port=60000:
DISCONNECTING
>>>>> client
>>>>>>>>>>> 192.168.6.148:43014 because read count=-1. Number
of
>> active
>>>>>>>>>> connections: 3
>>>>>>>>>>> 
>>>>>>>>>>> Do anyone has an idea about where this might
come from, or
>>> how
>>>> to
>>>>>>>>>> solve it
>>>>>>>>>>> ? Because I couldn't find much documentation
about this.
>>>>>>>>>>> Thanks in advance for your help !
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Loïc
>>>>>>>>>>> 
>>>>>>>>>>> Loïc CHANEL
>>>>>>>>>>> Engineering student at TELECOM Nancy
>>>>>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Thanks & Regards,
>>>>>> Anil Gupta
>> 
>> 
>> 
>> --
>> Thanks & Regards,
>> Anil Gupta
>> 

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message