spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Spark + HBase + Kerberos
Date Wed, 18 Mar 2015 18:46:47 GMT
Are hbase config / keytab files deployed on executor machines ?

Consider adding -Dsun.security.krb5.debug=true for debug purpose.

Cheers

On Wed, Mar 18, 2015 at 11:39 AM, Eric Walk <Eric.Walk@perficient.com>
wrote:

>  Having an issue connecting to HBase from a Spark container in a Secure
> Cluster. Haven’t been able to get past this issue, any thoughts would be
> appreciated.
>
>
>
> We’re able to perform some operations like “CreateTable” in the driver
> thread successfully. Read requests (always in the executor threads) are
> always failing with:
>
> No valid credentials provided (Mechanism level: Failed to find any
> Kerberos tgt)]
>
>
>
> Logs and scala are attached, the names of the innocent have masked for
> their protection (in a consistent manner).
>
>
>
> Executing the following spark job (using HDP 2.2, Spark 1.2.0, HBase
> 0.98.4, Kerberos on AD):
>
> export
> SPARK_CLASSPATH=/usr/hdp/2.2.0.0-2041/hbase/lib/hbase-server.jar:/usr/hdp/2.2.0.0-2041/hbase/lib/hbase-protocol.jar:/usr/hdp/2.2.0.0-2041/hbase/lib/hbase-hadoop2-compat.jar:/usr/hdp/2.2.0.0-2041/hbase/lib/hbase-client.jar:/usr/hdp/2.2.0.0-2041/hbase/lib/hbase-common.jar:/usr/hdp/2.2.0.0-2041/hbase/lib/htrace-core-3.0.4.jar:/usr/hdp/2.2.0.0-2041/hbase/lib/guava-12.0.1.jar:/usr/hdp/2.2.0.0-2041/hbase/conf
>
>
>
> /usr/hdp/2.2.0.0-2041/spark/bin/spark-submit --class HBaseTest
> --driver-memory 2g --executor-memory 1g --executor-cores 1 --num-executors
> 1 --master yarn-client ~/spark-test_2.10-1.0.jar
>
>
>
> We see this error in the executor processes (attached as yarn log.txt):
>
> 2015-03-18 17:34:15,121 DEBUG [Executor task launch worker-0]
> security.HBaseSaslRpcClient: Creating SASL GSSAPI client. Server's Kerberos
> principal name is hbase/ldevawshdp0002.<dc1>.pvc@<dc1>.PVC
>
> 2015-03-18 17:34:15,128 WARN  [Executor task launch worker-0]
> ipc.RpcClient: Exception encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
>
> 2015-03-18 17:34:15,129 ERROR [Executor task launch worker-0]
> ipc.RpcClient: SASL authentication failed. The most likely cause is missing
> or invalid credentials. Consider 'kinit'.
>
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
>
>
>
> The HBase Master Logs show success:
>
> 2015-03-18 17:34:12,861 DEBUG [RpcServer.listener,port=60000]
> ipc.RpcServer: RpcServer.listener,port=60000: connection from
> 10.4.0.6:46636; # active connections: 3
>
> 2015-03-18 17:34:12,872 DEBUG [RpcServer.reader=3,port=60000]
> ipc.RpcServer: Kerberos principal name is hbase/ldevawshdp0001.<dc1>.pvc@
> <DC1>.PVC
>
> 2015-03-18 17:34:12,875 DEBUG [RpcServer.reader=3,port=60000]
> ipc.RpcServer: Created SASL server with mechanism = GSSAPI
>
> 2015-03-18 17:34:12,875 DEBUG [RpcServer.reader=3,port=60000]
> ipc.RpcServer: Have read input token of size 1501 for processing by
> saslServer.evaluateResponse()
>
> 2015-03-18 17:34:12,876 DEBUG [RpcServer.reader=3,port=60000]
> ipc.RpcServer: Will send token of size 108 from saslServer.
>
> 2015-03-18 17:34:12,877 DEBUG [RpcServer.reader=3,port=60000]
> ipc.RpcServer: Have read input token of size 0 for processing by
> saslServer.evaluateResponse()
>
> 2015-03-18 17:34:12,878 DEBUG [RpcServer.reader=3,port=60000]
> ipc.RpcServer: Will send token of size 32 from saslServer.
>
> 2015-03-18 17:34:12,878 DEBUG [RpcServer.reader=3,port=60000]
> ipc.RpcServer: Have read input token of size 32 for processing by
> saslServer.evaluateResponse()
>
> 2015-03-18 17:34:12,879 DEBUG [RpcServer.reader=3,port=60000]
> security.HBaseSaslRpcServer: SASL server GSSAPI callback: setting
> canonicalized client ID: <user1>@<DC1>.PVC
>
> 2015-03-18 17:34:12,895 DEBUG [RpcServer.reader=3,port=60000]
> ipc.RpcServer: SASL server context established. Authenticated client:
> <user1>@<DC1>.PVC (auth:SIMPLE). Negotiated QoP is auth
>
> 2015-03-18 17:34:29,313 DEBUG [RpcServer.reader=3,port=60000]
> ipc.RpcServer: RpcServer.listener,port=60000: DISCONNECTING client
> 10.4.0.6:46636 because read count=-1. Number of active connections: 3
>
> 2015-03-18 17:34:37,102 DEBUG [RpcServer.listener,port=60000]
> ipc.RpcServer: RpcServer.listener,port=60000: connection from
> 10.4.0.6:46733; # active connections: 3
>
> 2015-03-18 17:34:37,102 DEBUG [RpcServer.reader=4,port=60000]
> ipc.RpcServer: RpcServer.listener,port=60000: DISCONNECTING client
> 10.4.0.6:46733 because read count=-1. Number of active connections: 3
>
>
>
> The Spark Driver Console Output hangs at this point:
>
> 2015-03-18 17:34:13,337 INFO  [main] spark.DefaultExecutionContext:
> Starting job: count at HBaseTest.scala:63
>
> 2015-03-18 17:34:13,349 INFO
> [sparkDriver-akka.actor.default-dispatcher-4] scheduler.DAGScheduler: Got
> job 0 (count at HBaseTest.scala:63) with 1 output partitions
> (allowLocal=false)
>
> 2015-03-18 17:34:13,350 INFO
> [sparkDriver-akka.actor.default-dispatcher-4] scheduler.DAGScheduler: Final
> stage: Stage 0(count at HBaseTest.scala:63)
>
> 2015-03-18 17:34:13,350 INFO
> [sparkDriver-akka.actor.default-dispatcher-4] scheduler.DAGScheduler:
> Parents of final stage: List()
>
> 2015-03-18 17:34:13,355 INFO
> [sparkDriver-akka.actor.default-dispatcher-4] scheduler.DAGScheduler:
> Missing parents: List()
>
> 2015-03-18 17:34:13,362 INFO
> [sparkDriver-akka.actor.default-dispatcher-4] scheduler.DAGScheduler:
> Submitting Stage 0 (NewHadoopRDD[0] at newAPIHadoopRDD at
> HBaseTest.scala:59), which has no missing parents
>
> 2015-03-18 17:34:13,377 INFO
> [sparkDriver-akka.actor.default-dispatcher-4] storage.MemoryStore:
> ensureFreeSpace(1680) called with curMem=394981, maxMem=1111794647
>
> 2015-03-18 17:34:13,378 INFO
> [sparkDriver-akka.actor.default-dispatcher-4] storage.MemoryStore: Block
> broadcast_1 stored as values in memory (estimated size 1680.0 B, free
> 1059.9 MB)
>
> 2015-03-18 17:34:13,380 INFO
> [sparkDriver-akka.actor.default-dispatcher-4] storage.MemoryStore:
> ensureFreeSpace(1165) called with curMem=396661, maxMem=1111794647
>
> 2015-03-18 17:34:13,381 INFO
> [sparkDriver-akka.actor.default-dispatcher-4] storage.MemoryStore: Block
> broadcast_1_piece0 stored as bytes in memory (estimated size 1165.0 B, free
> 1059.9 MB)
>
> 2015-03-18 17:34:13,382 INFO
> [sparkDriver-akka.actor.default-dispatcher-2] storage.BlockManagerInfo:
> Added broadcast_1_piece0 in memory on ldevawshdp0001.<dc1>.pvc:36023 (size:
> 1165.0 B, free: 1060.2 MB)
>
> 2015-03-18 17:34:13,384 INFO
> [sparkDriver-akka.actor.default-dispatcher-4] storage.BlockManagerMaster:
> Updated info of block broadcast_1_piece0
>
> 2015-03-18 17:34:13,392 INFO
> [sparkDriver-akka.actor.default-dispatcher-4]
> spark.DefaultExecutionContext: Created broadcast 1 from broadcast at
> DAGScheduler.scala:838
>
> 2015-03-18 17:34:13,408 INFO
> [sparkDriver-akka.actor.default-dispatcher-4] scheduler.DAGScheduler:
> Submitting 1 missing tasks from Stage 0 (NewHadoopRDD[0] at newAPIHadoopRDD
> at HBaseTest.scala:59)
>
> 2015-03-18 17:34:13,409 INFO
> [sparkDriver-akka.actor.default-dispatcher-4]
> cluster.YarnClientClusterScheduler: Adding task set 0.0 with 1 tasks
>
> 2015-03-18 17:34:13,416 INFO
> [sparkDriver-akka.actor.default-dispatcher-4] util.RackResolver: Resolved
> ldevawshdp0002.<dc1>.pvc to /default-rack
>
> 2015-03-18 17:34:13,428 INFO
> [sparkDriver-akka.actor.default-dispatcher-2] scheduler.TaskSetManager:
> Starting task 0.0 in stage 0.0 (TID 0, ldevawshdp0001.<dc1>.pvc,
> RACK_LOCAL, 1384 bytes)
>
> 2015-03-18 17:34:13,846 INFO
> [sparkDriver-akka.actor.default-dispatcher-2] storage.BlockManagerInfo:
> Added broadcast_1_piece0 in memory on ldevawshdp0001.<dc1>.pvc:50375 (size:
> 1165.0 B, free: 530.3 MB)
>
> 2015-03-18 17:34:13,993 INFO
> [sparkDriver-akka.actor.default-dispatcher-4] storage.BlockManagerInfo:
> Added broadcast_0_piece0 in memory on ldevawshdp0001.<dc1>.pvc:50375 (size:
> 53.2 KB, free: 530.2 MB)
>
>
>
> *Eric Walk*, Sr. Technical Consultant
> p: 617.855.9255 |  NASDAQ: PRFT  |  *Perficient.com
> <http://www.perficient.com/>*
>
> *[image: cid:image002.jpg@01CE554B.9F51CF10]* <http://www.perficient.com/>
>
> *[image: cid:image003.gif@01CE554B.9F51CF10]*
>
> *[image: cid:image004.jpg@01CE554B.9F51CF10]*
> <http://www.linkedin.com/groups/Perficient-Inc-78909?home=&gid=78909&trk=anet_ug_hm>*[image:
> cid:image005.jpg@01CE554B.9F51CF10]* <http://www.twitter.com/perficient>*[image:
> cid:image006.jpg@01CE554B.9F51CF10]*
> <http://www.perficient.com/Thought-Leadership/Social-Media/Blogs>*[image:
> cid:image007.jpg@01CE554B.9F51CF10]* <http://www.facebook.com/perficient>*[image:
> cid:image008.jpg@01CE554B.9F51CF10]* <http://www.youtube.com/perficient>
>
> *[image: cid:image009.jpg@01CE554B.9F51CF10]*
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>

Mime
View raw message