phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "gejx (JIRA)" <j...@apache.org>
Subject [jira] [Created] (PHOENIX-5198) GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
Date Fri, 15 Mar 2019 06:14:00 GMT
gejx created PHOENIX-5198:
-----------------------------

             Summary: GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)
                 Key: PHOENIX-5198
                 URL: https://issues.apache.org/jira/browse/PHOENIX-5198
             Project: Phoenix
          Issue Type: Bug
    Affects Versions: 5.0.0
         Environment: * 
>HDP 3.0.0

>Phoenix 5.0.0

>HBase 2.0.0

>Spark 2.3.1

>Hadoop 3.0.1
 
            Reporter: gejx
         Attachments: application_1551919460625_0204.txt

I re-run the program, the code is as follows:

code
{code:java}
@transient val confWrap = new Configuration()
confWrap.set("hbase.zookeeper.quorum", missionSession.config.zkQuorum)
confWrap.set("zookeeper.znode.parent", "/hbase-secure")
confWrap.set("hbase.zookeeper.property.clientPort", "2181")
confWrap.set("hadoop.security.authentication", "kerberos")
confWrap.set("hbase.security.authentication", "kerberos")
confWrap.set("hbase.myclient.keytab", missionSession.config.keytab)
confWrap.set("hbase.myclient.principal", missionSession.config.principal)

@transient val ugi: UserGroupInformation = UserGroupInformation.loginUserFromKeytabAndReturnUGI(missionSession.config.principal,
missionSession.config.keytab)
ugi.doAs(new PrivilegedExceptionAction[Unit] {
override def run(): Unit = {
val df: DataFrame = sqlContext.phoenixTableAsDataFrame(missionSession.config.tableName, Seq("ID",
"NAME"), zkUrl = Some(missionSession.config.zkUrl), conf = confWrap)
df.show(2)
}
}){code}

 
The parameters I submitted are as follows:
{code:java}
spark-submit --master yarn --name PHOENIX_SPARK_PLUGIN --deploy-mode cluster --driver-memory
1024M --executor-memory 1024M --num-executors 2 --executor-cores 1 --keytab /path/testdmp.keytab
--principal dmp@TESTDIP.ORG --conf spark.yarn.maxAppAttempts=1 --conf spark.driver.extraJavaOptions="-Dlog4j.configuration=log4j.properties"
--conf spark.executor.extraJavaOptions="-Dlog4j.configuration=log4j.properties" /opt/workspace/plugin/phoenix-spark-plugin-example-1.11.0-SNAPSHOT-jar-with-dependencies.jar
"DMP_CONF={\"spark\":{\"sparkMaster\":\"yarn\"},\"zkUrl\":\"jdbc:phoenix:test-dmp5.fengdai.org,test-dmp3.fengdai.org,test-dmp4.fengdai.org\",\"tableName\":\"DMP.DMP_TEST\"
,\"isDS\":true,\"zkQuorum\":\"test-dmp5.fengdai.org,test-dmp3.fengdai.org,test-dmp4.fengdai.org\",\"keytab\":\"/path/testdmp.keytab\",\"principal\":\"dmp@TESTDIP.ORG\"}"{code}
 

I tried to add keytab information to the url, but that didn't work. By reading the source
code, the keytab information is retrieved from conf when the login is checked. So I configured
it accordingly:

The conf for the sample:
{code:java}
confWrap.set("hbase.myclient.keytab", missionSession.config.keytab)
confWrap.set("hbase.myclient.principal", missionSession.config.principal){code}

 The url for the sample:
{code:java}
jdbc:phoenix:test-dmp5.fengdai.org,test-dmp3.fengdai.org,test-dmp4.fengdai.org:dmp@TESTDIP.ORG:/path/testdmp.keytab{code}
The submission parameter contains keytab information, driver can parse SQL,Excutor performed
the re-login operation, but still threw the exception GSSException,The excutor log shows "PrivilegedAction
as DMP ". Why does relogin not change the current UGI?

 

driver-log:
{code:java}
DEBUG UserGroupInformation: hadoop login
DEBUG UserGroupInformation: hadoop login commit
DEBUG UserGroupInformation: using local user:UnixPrincipal: dmp
DEBUG UserGroupInformation: Using user: "UnixPrincipal: dmp" with name dmp
DEBUG UserGroupInformation: User entry: "dmp"
DEBUG UserGroupInformation: Reading credentials from location set in HADOOP_TOKEN_FILE_LOCATION:
/hadoop/yarn/local/usercache/dmp/appcache/application_1551919460625_0199/container_e27_1551919460625_0199_01_000001/container_tokens
DEBUG UserGroupInformation: Loaded 3 tokens
DEBUG UserGroupInformation: UGI loginUser:dmp (auth:SIMPLE)
DEBUG UserGroupInformation: hadoop login
DEBUG UserGroupInformation: hadoop login commit
DEBUG UserGroupInformation: using kerberos user:dmp@TESTDIP.ORG
DEBUG UserGroupInformation: Using user: "dmp@TESTDIP.ORG" with name dmp@TESTDIP.ORG
DEBUG UserGroupInformation: User entry: "dmp@TESTDIP.ORG"
INFO UserGroupInformation: Login successful for user dmp@TESTDIP.ORG using keytab file testdmp.keytab-fb56007a-7d7d-4639-bf9e-5726b91901fd
DEBUG UserGroupInformation: PrivilegedAction as:dmp@TESTDIP.ORG (auth:KERBEROS) from:org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:814)
DEBUG UserGroupInformation: PrivilegedAction as:dmp@TESTDIP.ORG (auth:KERBEROS) from:org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:814)
{code}
excutor-log:
{code:java}
19/03/14 22:10:08 DEBUG SparkHadoopUtil: creating UGI for user: dmp
19/03/14 22:10:08 DEBUG UserGroupInformation: hadoop login
19/03/14 22:10:08 DEBUG UserGroupInformation: hadoop login commit
19/03/14 22:10:08 DEBUG UserGroupInformation: using local user:UnixPrincipal: dmp
19/03/14 22:10:08 DEBUG UserGroupInformation: Using user: "UnixPrincipal: dmp" with name dmp
19/03/14 22:10:08 DEBUG UserGroupInformation: User entry: "dmp"
19/03/14 22:10:08 DEBUG UserGroupInformation: Reading credentials from location set in HADOOP_TOKEN_FILE_LOCATION:
/hadoop/yarn/local/usercache/dmp/appcache/application_1551919460625_0204/container_e27_1551919460625_0204_01_000002/container_tokens
19/03/14 22:10:08 DEBUG UserGroupInformation: Loaded 3 tokens
19/03/14 22:10:08 DEBUG UserGroupInformation: UGI loginUser:dmp (auth:SIMPLE)
19/03/14 22:10:08 DEBUG UserGroupInformation: PrivilegedAction as:dmp (auth:SIMPLE) from:org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:64)
-----------------------------------------------------------------------------------------------------------------------------------------
19/03/14 22:10:50 DEBUG UserGroupInformation: hadoop login
19/03/14 22:10:50 DEBUG UserGroupInformation: hadoop login commit
19/03/14 22:10:50 DEBUG UserGroupInformation: using kerberos user:dmp@TESTDIP.ORG
19/03/14 22:10:50 DEBUG UserGroupInformation: Using user: "dmp@TESTDIP.ORG" with name dmp@TESTDIP.ORG
19/03/14 22:10:50 DEBUG UserGroupInformation: User entry: "dmp@TESTDIP.ORG"
19/03/14 22:10:50 INFO UserGroupInformation: Login successful for user dmp@TESTDIP.ORG using
keytab file /tesdmp/keytabs/nnjKorRc37PPPjLf/dmp/testdmp.keytab
------------------------------------------------------------------------------------------------------------------------------------------
19/03/14 22:11:02 DEBUG AbstractHBaseSaslRpcClient: Creating SASL GSSAPI client. Server's
Kerberos principal name is hbase/test-dmp4.fengdai.org@TESTDIP.ORG
19/03/14 22:11:03 DEBUG UserGroupInformation: PrivilegedAction as:dmp (auth:SIMPLE) from:org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.handlerAdded(NettyHBaseSaslRpcClientHandler.java:106)
19/03/14 22:11:03 DEBUG UserGroupInformation: PrivilegedActionException as:dmp (auth:SIMPLE)
cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid
credentials provided (Mechanism level: Failed to find any Kerberos tgt)]


{code}
In this method, relogging does not change current User, ConnectionInfo is cached based on
current User, and the connection is not available at this point:

 

log:
{code:java}
19/03/14 22:10:51 DEBUG PhoenixDriver: tmp==my current user is dmp (auth:SIMPLE)
19/03/14 22:10:51 DEBUG PhoenixDriver: tmp==my login user is dmp@TESTDIP.ORG (auth:KERBEROS){code}
method:
{code:java}
public ConnectionInfo normalize(ReadOnlyProps props, Properties info) throws SQLException
{
String zookeeperQuorum = this.getZookeeperQuorum();
Integer port = this.getPort();
String rootNode = this.getRootNode();
String keytab = this.getKeytab();
String principal = this.getPrincipal();
// Normalize connInfo so that a url explicitly specifying versus implicitly inheriting
// the default values will both share the same ConnectionQueryServices.
if (zookeeperQuorum == null) {
zookeeperQuorum = props.get(QueryServices.ZOOKEEPER_QUORUM_ATTRIB);
if (zookeeperQuorum == null) {
throw new SQLExceptionInfo.Builder(SQLExceptionCode.MALFORMED_CONNECTION_URL)
.setMessage(this.toString()).build().buildException();
}
}

if (port == null) {
if (!isConnectionless) {
String portStr = props.get(QueryServices.ZOOKEEPER_PORT_ATTRIB);
if (portStr != null) {
try {
port = Integer.parseInt(portStr);
} catch (NumberFormatException e) {
throw new SQLExceptionInfo.Builder(SQLExceptionCode.MALFORMED_CONNECTION_URL)
.setMessage(this.toString()).build().buildException();
}
}
}
} else if (isConnectionless) {
throw new SQLExceptionInfo.Builder(SQLExceptionCode.MALFORMED_CONNECTION_URL)
.setMessage("Port may not be specified when using the connectionless url \"" + this.toString()
+ "\"").build().buildException();
}
if (rootNode == null) {
if (!isConnectionless) {
rootNode = props.get(QueryServices.ZOOKEEPER_ROOT_NODE_ATTRIB);
}
} else if (isConnectionless) {
throw new SQLExceptionInfo.Builder(SQLExceptionCode.MALFORMED_CONNECTION_URL)
.setMessage("Root node may not be specified when using the connectionless url \"" + this.toString()
+ "\"").build().buildException();
}
if (principal == null) {
if (!isConnectionless) {
principal = props.get(QueryServices.HBASE_CLIENT_PRINCIPAL);
}
}
if (keytab == null) {
if (!isConnectionless) {
keytab = props.get(QueryServices.HBASE_CLIENT_KEYTAB);
}
}
if (!isConnectionless()) {
boolean credsProvidedInUrl = null != principal && null != keytab;
boolean credsProvidedInProps = info.containsKey(QueryServices.HBASE_CLIENT_PRINCIPAL) &&
info.containsKey(QueryServices.HBASE_CLIENT_KEYTAB);
if (credsProvidedInUrl || credsProvidedInProps) {
// PHOENIX-3189 Because ConnectionInfo is immutable, we must make sure all parts of it are
correct before
// construction; this also requires the Kerberos user credentials object (since they are compared
by reference
// and not by value. If the user provided a principal and keytab via the JDBC url, we must
make sure that the
// Kerberos login happens *before* we construct the ConnectionInfo object. Otherwise, the
use of ConnectionInfo
// to determine when ConnectionQueryServices impl's should be reused will be broken.
try {
// Check if we need to authenticate with kerberos so that we cache the correct ConnectionInfo
UserGroupInformation currentUser = UserGroupInformation.getCurrentUser();
if (!currentUser.hasKerberosCredentials() || !isSameName(currentUser.getUserName(), principal))
{
synchronized (KERBEROS_LOGIN_LOCK) {
// Double check the current user, might have changed since we checked last. Don't want
// to re-login if it's the same user.
currentUser = UserGroupInformation.getCurrentUser();
if (!currentUser.hasKerberosCredentials() || !isSameName(currentUser.getUserName(), principal))
{
final Configuration config = getConfiguration(props, info, principal, keytab);
logger.info("Trying to connect to a secure cluster as {} with keytab {}", config.get(QueryServices.HBASE_CLIENT_PRINCIPAL),
config.get(QueryServices.HBASE_CLIENT_KEYTAB));
UserGroupInformation.setConfiguration(config);
User.login(config, QueryServices.HBASE_CLIENT_KEYTAB, QueryServices.HBASE_CLIENT_PRINCIPAL,
null);
logger.info("tmp==ugi user is{},auth is{}" ,UserGroupInformation.getCurrentUser().getUserName(),UserGroupInformation.getCurrentUser().getAuthenticationMethod());
logger.info("tmp==ugi login user is{},auth is{}" ,UserGroupInformation.getLoginUser().getUserName(),UserGroupInformation.getLoginUser().getAuthenticationMethod());
logger.info("Successful login to secure cluster");
}
}
} else {
// The user already has Kerberos creds, so there isn't anything to change in the ConnectionInfo.
logger.debug("Already logged in as {}", currentUser);
}
} catch (IOException e) {
throw new SQLExceptionInfo.Builder(SQLExceptionCode.CANNOT_ESTABLISH_CONNECTION)
.setRootCause(e).build().buildException();
}
} else {
logger.debug("Principal and keytab not provided, not attempting Kerberos login");
}
} // else, no connection, no need to login
// Will use the current User from UGI
return new ConnectionInfo(zookeeperQuorum, port, rootNode, principal, keytab);
}

 
{code}
So I always get the following exceptions: 


{code:java}
19/03/14 22:11:11 DEBUG ClientCnxn: Reading reply sessionid:0x26975f6aaa9056d, packet:: clientPath:/hbase-secure/meta-region-server
serverPath:/hbase-secure/meta-region-server finished:false header:: 9,4 replyHeader:: 9,34359750201,0
request:: '/hbase-secure/meta-region-server,F response:: #ffffffff000146d61737465723a3136303030ffffffa1dffffffbafffffff043fffffff53d7c50425546a21a15746573742d646d70342e66656e676461692e6f726710ffffff947d18ffffffe5fffffff6ffffffc1ffffffb0ffffff972d100183,s{8589935420,34359739604,1543999435222,1552464056849,257,0,0,0,68,0,8589935420}
19/03/14 22:11:11 DEBUG AbstractHBaseSaslRpcClient: Creating SASL GSSAPI client. Server's
Kerberos principal name is hbase/test-dmp4.fengdai.org@TESTDIP.ORG
19/03/14 22:11:11 DEBUG UserGroupInformation: PrivilegedAction as:dmp (auth:SIMPLE) from:org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.handlerAdded(NettyHBaseSaslRpcClientHandler.java:106)
19/03/14 22:11:11 DEBUG UserGroupInformation: PrivilegedActionException as:dmp (auth:SIMPLE)
cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid
credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
19/03/14 22:11:11 DEBUG RpcRetryingCallerImpl: Call exception, tries=7, retries=7, started=11862
ms ago, cancelled=false, msg=Call to test-dmp4.fengdai.org/10.200.162.25:16020 failed on local
exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)], details=row
'SYSTEM:CATALOG' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=test-dmp4.fengdai.org,16020,1552463985509,
seqNum=-1, exception=java.io.IOException: Call to test-dmp4.fengdai.org/10.200.162.25:16020
failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused
by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)]
at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)
at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)
at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)
at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)
at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:353)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)
at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)
at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929)
at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179)
at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71)
at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:247)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:163)
at org.apache.hadoop.hbase.ipc.NettyRpcConnection.saslNegotiate(NettyRpcConnection.java:201)
at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$800(NettyRpcConnection.java:71)
at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:273)
at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:261)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
at org.apache.hbase.thir
{code}

I uploaded a full debug log. Can anyone write a suggestion for me?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message