spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: spark-shell with different username
Date Sat, 02 Apr 2016 06:16:54 GMT
Matt,

What OS are you using on your laptop? Sounds like Ubuntu or something?

Thanks

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com



On 2 April 2016 at 01:17, Matt Tenenbaum <matt.tenenbaum@rockyou.com> wrote:

> Hello all —
>
> tl;dr: I’m having an issue running spark-shell from my laptop (or other
> non-cluster-affiliated machine), and I think the issue boils down to
> usernames. Can I convince spark/scala that I’m someone other than $USER?
>
> A bit of background: our cluster is CDH 5.4.8, installed with Cloudera
> Manager 5.5. We use LDAP, and my login on all hadoop-affiliated machines
> (including the gateway boxes we use for running scheduled work) is
> ‘matt.tenenbaum’. When I run spark-shell on one of those machines,
> everything is fine:
>
> [matt.tenenbaum@remote-machine ~]$ HADOOP_CONF_DIR=/etc/hadoop/conf SPARK_HOME=spark-1.6.0-bin-hadoop2.6
spark-1.6.0-bin-hadoop2.6/bin/spark-shell --master yarn --deploy-mode client
>
> Everything starts up correctly, I get a scala prompt, the SparkContext and
> SQL context are correctly initialized, and I’m off to the races:
>
> 16/04/01 23:27:00 INFO session.SessionState: Created local directory: /tmp/35b58974-dad5-43c6-9864-43815d101ca0_resources
> 16/04/01 23:27:00 INFO session.SessionState: Created HDFS directory: /tmp/hive/matt.tenenbaum/35b58974-dad5-43c6-9864-43815d101ca0
> 16/04/01 23:27:00 INFO session.SessionState: Created local directory: /tmp/matt.tenenbaum/35b58974-dad5-43c6-9864-43815d101ca0
> 16/04/01 23:27:00 INFO session.SessionState: Created HDFS directory: /tmp/hive/matt.tenenbaum/35b58974-dad5-43c6-9864-43815d101ca0/_tmp_space.db
> 16/04/01 23:27:00 INFO repl.SparkILoop: Created sql context (with Hive support)..
> SQL context available as sqlContext.
>
> scala> 1 + 41
> res0: Int = 42
>
> scala> sc
> res1: org.apache.spark.SparkContext = org.apache.spark.SparkContext@4e9bd2c8
>
> I am running 1.6 from a downloaded tgz file, rather than the spark-shell
> made available to the cluster from CDH. I can copy that tgz to my laptop,
> and grab a copy of the cluster configurations, and in a perfect world I
> would then be able to run everything in the same way
>
> [matt@laptop ~]$ HADOOP_CONF_DIR=path/to/hadoop/conf SPARK_HOME=spark-1.6.0-bin-hadoop2.6
spark-1.6.0-bin-hadoop2.6/bin/spark-shell --master yarn --deploy-mode client
>
> Notice there are two things that are different:
>
>    1. My local username on my laptop is ‘matt’, which does not match my
>    name on the remote machine.
>    2. The Hadoop configs live somewhere other than /etc/hadoop/conf
>
> Alas, #1 proves fatal because of cluster permissions (there is no
> /user/matt/ in HDFS, and ‘matt’ is not a valid LDAP user). In the
> initialization logging output, I can see that fail in an expected way:
>
> 16/04/01 16:37:19 INFO yarn.Client: Setting up container launch context for our AM
> 16/04/01 16:37:19 INFO yarn.Client: Setting up the launch environment for our AM container
> 16/04/01 16:37:19 INFO yarn.Client: Preparing resources for our AM container
> 16/04/01 16:37:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for
your platform... using builtin-java classes where applicable
> 16/04/01 16:37:21 ERROR spark.SparkContext: Error initializing SparkContext.
> org.apache.hadoop.security.AccessControlException: Permission denied: user=matt, access=WRITE,
inode="/user":hdfs:supergroup:drwxr-xr-x
>     at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
>     at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
>     at (... etc ...)
>
> Fine. In other circumstances I’ve told Hadoop explicitly who I am by
> setting HADOOP_USER_NAME. Maybe that works here?
>
> [matt@laptop ~]$ HADOOP_USER_NAME=matt.tenenbaum HADOOP_CONF_DIR=soma-conf SPARK_HOME=spark-1.6.0-bin-hadoop2.6
spark-1.6.0-bin-hadoop2.6/bin/spark-shell --master yarn --deploy-mode client
>
> Eventually that fails too, but not for the same reason. Setting
> HADOOP_USER_NAME is sufficient to allow initialization to get past the
> access-control problems, and I can see it request a new application from
> the cluster
>
> 16/04/01 16:43:08 INFO yarn.Client: Will allocate AM container, with 896 MB memory including
384 MB overhead
> 16/04/01 16:43:08 INFO yarn.Client: Setting up container launch context for our AM
> 16/04/01 16:43:08 INFO yarn.Client: Setting up the launch environment for our AM container
> 16/04/01 16:43:08 INFO yarn.Client: Preparing resources for our AM container
> ... [resource uploads happen here] ...
> 16/04/01 16:46:16 INFO spark.SecurityManager: Changing view acls to: matt,matt.tenenbaum
> 16/04/01 16:46:16 INFO spark.SecurityManager: Changing modify acls to: matt,matt.tenenbaum
> 16/04/01 16:46:16 INFO spark.SecurityManager: SecurityManager: authentication disabled;
ui acls disabled; users with view permissions: Set(matt, matt.tenenbaum); users with modify
permissions: Set(matt, matt.tenenbaum)
> 16/04/01 16:46:16 INFO yarn.Client: Submitting application 30965 to ResourceManager
> 16/04/01 16:46:16 INFO impl.YarnClientImpl: Submitted application application_1451332794331_30965
> 16/04/01 16:46:17 INFO yarn.Client: Application report for application_1451332794331_30965
(state: ACCEPTED)
> 16/04/01 16:46:17 INFO yarn.Client:
>      client token: N/A
>      diagnostics: N/A
>      ApplicationMaster host: N/A
>      ApplicationMaster RPC port: -1
>      queue: root.matt_dot_tenenbaum
>      start time: 1459554373844
>      final status: UNDEFINED
>      tracking URL: http://resource-manager:8088/proxy/application_1451332794331_30965/
>      user: matt.tenenbaum
> 16/04/01 16:46:19 INFO yarn.Client: Application report for application_1451332794331_30965
(state: ACCEPTED)
>
> but this AM never switches state from ACCEPTED to RUNNING. Eventually it
> times out and kills the AM
>
> 16/04/01 16:50:14 INFO yarn.Client: Application report for application_1451332794331_30965
(state: FAILED)
> 16/04/01 16:50:14 INFO yarn.Client:
>      client token: N/A
>      diagnostics: Application application_1451332794331_30965 failed 2 times due to AM
Container for appattempt_1451332794331_30965_000002 exited with  exitCode: 10
> For more detailed output, check application tracking page:http://resource-manager:8088/proxy/application_1451332794331_30965/Then,
click on links to logs of each attempt.
> Diagnostics: Exception from container-launch.
> Container id: container_e43_1451332794331_30965_02_000001
> Exit code: 10
> Stack trace: ExitCodeException exitCode=10:
>     at org.apache.hadoop.util.Shell.runCommand(Shell.java:543)
>     at org.apache.hadoop.util.Shell.run(Shell.java:460)
>     at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:720)
>     at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:293)
>     at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
>     at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>     at java.lang.Thread.run(Thread.java:745)
>
> Shell output: main : command provided 1
> main : user is yarn
> main : requested yarn user is matt.tenenbaum
>
> Container exited with a non-zero exit code 10
> Failing this attempt. Failing the application.
>      ApplicationMaster host: N/A
>      ApplicationMaster RPC port: -1
>      queue: root.matt_dot_tenenbaum
>      start time: 1459554373844
>      final status: FAILED
>      tracking URL: http://resource-manager:8088/cluster/app/application_1451332794331_30965
>      user: matt.tenenbaum
> 16/04/01 16:50:15 ERROR spark.SparkContext: Error initializing SparkContext.
> org.apache.spark.SparkException: Yarn application has already ended! It might have been
killed or unable to launch application master.
>     at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:124)
>     at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:64)
>     at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
>     at org.apache.spark.SparkContext.<init>(SparkContext.scala:530)
>     at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017)
>     at $line3.$read$$iwC$$iwC.<init>(<console>:15)
>     at $line3.$read$$iwC.<init>(<console>:24)
>     at $line3.$read.<init>(<console>:26)
>     at $line3.$read$.<init>(<console>:30)
>     at $line3.$read$.<clinit>(<console>)
>     at $line3.$eval$.<init>(<console>:7)
>     at $line3.$eval$.<clinit>(<console>)
>     at $line3.$eval.$print(<console>)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:606)
>     at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
>     at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)
>     at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
>     at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
>     at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
>     at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
>     at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
>     at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
>     at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:125)
>     at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124)
>     at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
>     at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124)
>     at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
>     at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)
>     at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159)
>     at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
>     at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108)
>     at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
>     at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991)
>     at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
>     at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
>     at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
>     at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
>     at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
>     at org.apache.spark.repl.Main$.main(Main.scala:31)
>     at org.apache.spark.repl.Main.main(Main.scala)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:606)
>     at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
>     at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
>     at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
>     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
>     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> In the end, I’m left at a scala prompt but (obviously) without sc or
> sqlContext
>
> <console>:16: error: not found: value sqlContext
>          import sqlContext.implicits._
>                 ^
> <console>:16: error: not found: value sqlContext
>          import sqlContext.sql
>                 ^
>
> scala>
>
> A bit of googling and reading on Stack Overflow suggests that this all
> boils down to the SecurityManager, and the difference between running on
> remote where the shell user matches the expected Hadoop user (so
> scala.SecurityManager sees Set(matt.tenenbaum)) vs running on my laptop
> where the SecurityManager sees Set(matt, matt.tenenbaum). I tried
> manually setting the SPARK_IDENT_STRING and USER environment variables to
> “matt.tenenbaum” also, but that doesn’t change the outcome.
>
> Am I even on the right track? Is this because of a mismatch between who I
> am on my laptop and who the cluster wants me to be? Is there any way to
> convince my local spark-shell invocation that I’m “matt.tenenbaum”, not
> “matt”?
>
> Thank you for reading this far, and for any suggestions
> -mt
> ​
>

Mime
View raw message