hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saad Mufti <saad.mu...@gmail.com>
Subject Re: HBase Replication Between Two Secure Clusters With Different Kerberos KDC's
Date Wed, 23 May 2018 15:31:11 GMT
Thanks, here it is:

1. On the source cluster, will be identical on the target cluster as both
use the same Kerberos realm name, though each has its own cluster specific
KDC:

$ more /etc/zookeeper/conf/server-jaas.conf
>
> /**
>
>  * Licensed to the Apache Software Foundation (ASF) under one
>
>  * or more contributor license agreements.  See the NOTICE file
>
>  * distributed with this work for additional information
>
>  * regarding copyright ownership.  The ASF licenses this file
>
>  * to you under the Apache License, Version 2.0 (the
>
>  * "License"); you may not use this file except in compliance
>
>  * with the License.  You may obtain a copy of the License at
>
>  * <p/>
>
>  * http://www.apache.org/licenses/LICENSE-2.0
>
>  * <p/>
>
>  * Unless required by applicable law or agreed to in writing, software
>
>  * distributed under the License is distributed on an "AS IS" BASIS,
>
>  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> implied.
>
>  * See the License for the specific language governing permissions and
>
>  * limitations under the License.
>
>  */
>
> Server {
>
>       com.sun.security.auth.module.Krb5LoginModule required
>
>       useKeyTab=true
>
>       keyTab="/etc/zookeeper.keytab"
>
>       storeKey=true
>
>       useTicketCache=false
>
>       principal="zookeeper/<source-master-ip>@PGS.dev";
>
> };
>
>
2. I ran zkCli.sh after authenticating as kerberos principal
zookeeper/<host>@PGS.dev got the following:

getAcl /hbase
>
> 'world,'anyone
>
> : r
>
> 'sasl,'hbase
>
> : cdrwa
>
>
3. I was logged in as Kerberos principal hbase/<host>@PGS.dev when I ran
the add_peer command

Thanks for taking the time to help me in any way you can.

----
Saad


On Wed, May 23, 2018 at 7:24 AM, Reid Chan <reidddchan@outlook.com> wrote:

> Three places to check,
>
>
>   1.  Would you mind showing your "/etc/zookeeper/conf/server-jaas.conf",
>
>     2. and using zkCli.sh to getAcl /hbase.
>     3. BTW, what was your login principal when executing "add_peer" in
> hbase shell.
> ________________________________
> From: Saad Mufti <saad.mufti@gmail.com>
> Sent: 23 May 2018 01:48:17
> To: user@hbase.apache.org
> Subject: HBase Replication Between Two Secure Clusters With Different
> Kerberos KDC's
>
> Hi,
>
> Here is my scenario, I have two secure/authenticated EMR based HBase
> clusters, both have their own cluster dedicated KDC (using EMR support for
> this which means we get Kerberos support by just turning on a config flag).
>
> Now we want to get replication going between them. For other application
> reasons, we want both clusters to have the same Kerberos realm, let's say
> APP.COM, so Kerberos principals are like api@APP.COM .
>
> I looked around the web and found the instructions at
> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/
> bk_hadoop-high-availability/content/hbase-cluster-
> replication-among-kerberos-secured-clusters.html
> so I tried to follow these directions. Of course the instructions are for
> replication between clusters with different realms, so I adapted this by
> adding only one principal "krbtgt/APP.COM@APP.COM" and gave it some
> arbitrary password. Followed the rest of the directions as well to pass a
> rule property to Zookeeper and the requisite Hadoop property in
> core-site.xml .
>
> After all this, when I set up replication from cluster1 to cluster using
> add_peer, I see error messages in the region servers for cluster1 of the
> following form:
>
>
>
> > 2018-05-22 17:27:45,763 INFO  [main-SendThread(xxx.net:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server i
> >
> > p-10-194-247-88.aolp-ds-dev.us-east-1.ec2.aolcloud.net/xxx.yyy.zzz:2181.
> > Will attempt to SASL-authenticate using Login Context section 'Client'
> >
> > 2018-05-22 17:27:45,764 INFO  [main-SendThread(xxx.net:2181)]
> > zookeeper.ClientCnxn: Socket connection established to ip-1
> >
> > 0-194-247-88.aolp-ds-dev.us-east-1.ec2.aolcloud.net/xxx.yyy.zzz:2181,
> > initiating session
> >
> > 2018-05-22 17:27:45,777 INFO  [main-SendThread(xxx.net:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on ser
> >
> > ver xxx.net/xxx.yyy.zzz:2181, sessionid = 0x16388599b300215, negotiated
> > timeout = 40000
> > 2018-05-22 17:27:45,779 ERROR [main-SendThread(xxx.net:2181)]
> > client.ZooKeeperSaslClient: An error: (java.security.Privil
> >
> > egedActionException: javax.security.sasl.SaslException: GSS initiate
> > failed [Caused by GSSException: No valid credentials provided (Mechanism
> > level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)])
> > occurred when evaluating Zookeeper Quorum Member's  received SASL token.
> > Zookeeper Client will go to AUTH_FAILED state.
> >
> > 2018-05-22 17:27:45,779 ERROR [main-SendThread(xxx.net:2181)]
> > zookeeper.ClientCnxn: SASL authentication with Zookeeper Quorum member
> > failed: javax.security.sasl.SaslException: An error:
> > (java.security.PrivilegedActionException:
> > javax.security.sasl.SaslException: GSS initiate failed [Caused by
> > GSSException: No valid credentials provided (Mechanism level: Server not
> > found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
> > evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper
> > Client will go to AUTH_FAILED state.
> >
> > 2018-05-22 17:28:12,574 WARN  [main-EventThread] zookeeper.ZKUtil:
> > hconnection-0x4dcc1d33-0x16388599b300215, quorum=xyz.net:2181,
> > baseZNode=/hbase Unable to set watcher on znode (/hbase/hbaseid)
> >
> > org.apache.zookeeper.KeeperException$AuthFailedException:
> KeeperErrorCode
> > = AuthFailed for /hbase/hbaseid
> >
> >         at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
> >
> >         at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >
> >         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
> >
> >         at
> >
> >
> The Zookeeper start command looks like the following:
>
> /usr/lib/jvm/java-openjdk/bin/java -D*zoo*keeper.log.dir=/var/
> log/*zoo*keeper
> > -D*zoo*keeper.root.logger=INFO,ROLLINGFILE -cp /usr/lib/*zoo*
> > keeper/bin/../build/classes:/usr/lib/*zoo*
> > keeper/bin/../build/lib/*.jar:/usr/lib/*zoo*
> > keeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/lib/*zoo*
> > keeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/lib/*zoo*
> > keeper/bin/../lib/netty-3.10.5.Final.jar:/usr/lib/*zoo*
> > keeper/bin/../lib/log4j-1.2.16.jar:/usr/lib/*zoo*
> > keeper/bin/../lib/jline-2.11.jar:/usr/lib/*zoo*keeper/bin/../*zoo*
> > keeper-3.4.10.jar:/usr/lib/*zoo*keeper/bin/../src/java/lib/*.jar:/etc/
> > *zoo*keeper/conf::/etc/*zoo*keeper/conf:/usr/lib/*zoo*keeper/*:/usr/lib/
> > *zoo*keeper/lib/* -Djava.security.auth.login.
> config=/etc/*zoo*keeper/conf/server-jaas.conf
> > -D*zoo*keeper.security.auth_to_local=RULE:[2:\$1@\$0](.*@\QAPP.COM
> \E$)s/@\
> > APP.COM\E$//DEFAULT -*zoo*keeper.log.threshold=INFO
> > -Dcom.sun.management.jmxremote
> > -Dcom.sun.management.jmxremote.local.only=false
> org.apache.*zoo*keeper.server.quorum.QuorumPeerMain
> > /etc/*zoo*keeper/conf/*zoo*.cfg
> >
> >
> The property in core-site looks like the following:
>
>   <property>
> >
> >     <name>hadoop.security.auth_to_local</name>
> >
> >     <value>RULE:[2:\$1@
> > \$0](.*@\\QPGS.dev\\E$)s/@\\QPGS.dev\\E$//DEFAULT</value>
> >
> >   </property>
> >
>
>
>
> At this point I am not clear how I can get the added Kerberos principal  "
> krbtgt/APP.COM@APP.COM" (in both clusters' Kerberos KDC's) to be
> authenticated against and for replication to start flowing.
>
> Any help and/or pointers would be appreciated.
>
> Thanks.
>
> -----
> Saad
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message