hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Kerberos exception
Date Sun, 15 Mar 2015 21:43:05 GMT
Looks like region server log didn't go through.

Can you use pastebin ?

On Sun, Mar 15, 2015 at 10:04 AM, Manoj Murumkar <manoj.murumkar@gmail.com>
wrote:

> Attached region server log again.
>
> Ted,
>
> Corresponding log for master is:
>
> java.io.IOException: Couldn't setup connection for hbase/
> sfdvgctmn001.gid.gap.com@SFDVGCT.COM to hbase/sfdvgctsn001.gid.gap.com@S
>
> FDVGCT.COM
>
>         at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
>
>         at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)
>
>         at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:21176)
>
>         at
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:670)
>
>         at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2004)
>
>         at
> org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManager.java:119)
>
>         at
> org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager.java:743)
>
>         at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
>         at java.lang.Thread.run(Thread.java:745)
>
> Caused by:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> GSS initiate failed
>
>         at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151)
>
>         at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:187)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)
>
> On Sun, Mar 15, 2015 at 9:36 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
>> bq. Attached both files
>>
>> I only found hbase-site.xml in the attachment.
>>
>> On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar <manoj.murumkar@gmail.com
>> >
>> wrote:
>>
>> > Attached both files.
>> >
>> >
>> >
>> > On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov <olorinbant@gmail.com>
>> > wrote:
>> >
>> >> Hi,
>> >>
>> >> Traces (especially one for region server) look a bit incomplete, did
>> >> you copy them fully?
>> >>
>> >> Also may help if you post relevant pieces of hbase-site.xml (with
>> >> security configs).
>> >>
>> >> Thanks,
>> >> Mikhail
>> >>
>> >>
>> >> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar
>> >> <manoj.murumkar@gmail.com> wrote:
>> >> > Hi,
>> >> >
>> >> > We have a secured cluster. All components are working well, except
>> >> hbase.
>> >> > Specifically, this is what I see on regionserver:
>> >> >
>> >> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
>> >> > ipc.RpcServer: Kerberos principal name is hbase/
>> >> > sfdvgctsn001.xxxxx@SFDVGCT.COM
>> >> >
>> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>> >> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
>> >> >
>> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>> >> > ipc.RpcServer: Have read input token of size 627 for processing by
>> >> > saslServer.evaluateResponse()
>> >> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
>> >> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while
>> >> > reading:GSS initiate fail
>> >> >
>> >> >
>> >> > and on the master:
>> >> >
>> >> >
>> >> > java.io.IOException: Couldn't setup connection for
>> hbase/sfdvgctmn004.
>> >> > xxxxx@SFDVGCT.COM to hbase/sfdvgctsn001.xxxxx@SFDVGCT.COM
>> >> >
>> >> >         at
>> >> >
>> >>
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
>> >> >
>> >> >         at java.security.AccessController.doPrivileged(Native Method)
>> >> >
>> >> >         at javax.security.auth.Subject.doAs(Subject.java:415)
>> >> >
>> >> >         at
>> >> >
>> >>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>> >> >
>> >> >         at
>> >> >
>> >>
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
>> >> >
>> >> >         at
>> >> >
>> >>
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
>> >> >
>> >> >         at
>> >> >
>> >>
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
>> >> >
>> >> >         at
>> >> >
>> >>
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
>> >> >
>> >> > All the keytabs are set properly. Has anyone seen this before?
>> >> Appreciate
>> >> > the time.
>> >> >
>> >> > Manoj
>> >>
>> >>
>> >>
>> >> --
>> >> Thanks,
>> >> Michael Antonov
>> >>
>> >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message