spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ajith shetty <ajith.she...@huawei.com>
Subject [Spark][Security] UGI credentials lost between driver and executor in yarn mode
Date Wed, 21 Mar 2018 05:50:58 GMT
Hi all

I see UGI credentials (ex sparkCookie) shared from driver to executor is being lost on driver
side in yarn mode. Below is the analysis on start of thriftserver,

Step 1. SparkSubmit create submit env which does a loginUserFromKeytab
  "main@1" prio=5 tid=0x1 nid=NA runnable
    java.lang.Thread.State: RUNNABLE
                  at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1041)
                  - locked <0x582> (a java.lang.Class)
                  at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:336)
                  at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:156)
                  at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:122)
                  at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala:-1)

Step 2. HiveThriftServer does SparkSQLEnv.init which will Create SparkContext, and hence calls
createDriverEnv which will generate secret key and add to UGI by Step 1 as credentials
                "main@1" prio=5 tid=0x1 nid=NA runnable
                java.lang.Thread.State: RUNNABLE
                                at org.apache.spark.SecurityManager.generateSecretKey(SecurityManager.scala:429)
                                at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:228)
                                at org.apache.spark.SparkEnv$.create(SparkEnv.scala:237)
                                at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175)
                                at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:258)
                                at org.apache.spark.SparkContext.<init>(SparkContext.scala:433)
                                at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2521)
                                - locked <0x10a5> (a java.lang.Object)
                                at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:923)
                                at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:915)
                                at scala.Option.getOrElse(Option.scala:121)
                                at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:915)
                                - locked <0x1091> (a org.apache.spark.sql.SparkSession$Builder)
                                - locked <0x10a6> (a org.apache.spark.sql.SparkSession$)
                                at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:48)
                                at org.apache.spark.sql.hive.thriftserver.a$.main(HiveThriftServer2.scala:86)
                                at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala:-1)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                                at java.lang.reflect.Method.invoke(Method.java:498)
                                at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:798)
                                at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:183)
                                at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:208)
                                at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:122)
                                at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala:-1)


Step 3. Next Application is submitted which will create container launch context using UGI
passed from Step 2
                "main@1" prio=5 tid=0x1 nid=NA runnable
                java.lang.Thread.State: RUNNABLE
                                at org.apache.spark.SecurityManager.generateSecretKey(SecurityManager.scala:429)
                                at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:228)
                                at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:999)
                                at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:194)
                                at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
                                at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)
                                at org.apache.spark.SparkContext.<init>(SparkContext.scala:510)
                                at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2521)
                                - locked <0x10a5> (a java.lang.Object)
                                at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:923)
                                at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:915)
                                at scala.Option.getOrElse(Option.scala:121)
                                at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:915)
                                - locked <0x1091> (a org.apache.spark.sql.SparkSession$Builder)
                                - locked <0x10a6> (a org.apache.spark.sql.SparkSession$)
                                at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:48)
                                at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:86)
                                at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala:-1)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                                at java.lang.reflect.Method.invoke(Method.java:498)
                                at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:798)
                                at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:183)
                                at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:208)
                                at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:122)
                                at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala:-1)

Step 4. Lastly initCompositeService will invoke SparkSQLCLIService which does a loginUserFromKeytab
again which will override UGI created by Step 1
                "main@1" prio=5 tid=0x1 nid=NA runnable
                java.lang.Thread.State: RUNNABLE
                                at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1041)
                                - locked <0x582> (a java.lang.Class)
                                at org.apache.hive.service.auth.HiveAuthFactory.loginFromKeytab(HiveAuthFactory.java:199)
                                at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIService.init(SparkSQLCLIService.scala:53)
                                at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService$$anonfun$initCompositeService$1.apply(SparkSQLCLIService.scala:79)
                                at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService$$anonfun$initCompositeService$1.apply(SparkSQLCLIService.scala:79)
                                at scala.collection.Iterator$class.foreach(Iterator.scala:893)
                                at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
                                at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
                                at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
                                at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService$class.initCompositeService(SparkSQLCLIService.scala:79)
                                at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.initCompositeService(HiveThriftServer2.scala:283)
                                at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.init(HiveThriftServer2.scala:303)
                                at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:99)
                                at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala:-1)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                                at java.lang.reflect.Method.invoke(Method.java:498)
                                at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:798)
                                at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:183)
                                at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:208)
                                at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:122)
                                at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala:-1)

Is this the right behavior? as it looks like UGI created on JDBCServer startup which is passed
to containerLaunchContext becomes stale.? please correct me if i am wrong

Regards
Ajith

Mime
View raw message