spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: ERROR UserGroupInformation: PriviledgedActionException
Date Wed, 05 Nov 2014 06:32:45 GMT
Its more like you are having different versions of spark

Thanks
Best Regards

On Wed, Nov 5, 2014 at 3:05 AM, Saiph Kappa <saiph.kappa@gmail.com> wrote:

> I set the host and port of the driver and now the error slightly changed
>
> Using Spark's default log4j profile:
>> org/apache/spark/log4j-defaults.properties
>> 14/11/04 21:13:48 INFO CoarseGrainedExecutorBackend: Registered signal
>> handlers for [TERM, HUP, INT]
>> 14/11/04 21:13:48 INFO SecurityManager: Changing view acls to:
>> myuser,Myuser
>> 14/11/04 21:13:48 INFO SecurityManager: Changing modify acls to:
>> myuser,Myuser
>> 14/11/04 21:13:48 INFO SecurityManager: SecurityManager: authentication
>> disabled; ui acls disabled; users with view permissions: Set(myuser,
>> Myuser); users with modify permissions: Set(myuser, Myuser)
>> 14/11/04 21:13:48 INFO Slf4jLogger: Slf4jLogger started
>> 14/11/04 21:13:48 INFO Remoting: Starting remoting
>> 14/11/04 21:13:49 INFO Remoting: Remoting started; listening on addresses
>> :[akka.tcp://driverPropsFetcher@myserver:37456]
>> 14/11/04 21:13:49 INFO Remoting: Remoting now listens on addresses:
>> [akka.tcp://driverPropsFetcher@myserver:37456]
>> 14/11/04 21:13:49 INFO Utils: Successfully started service
>> 'driverPropsFetcher' on port 37456.
>> 14/11/04 21:14:19 ERROR UserGroupInformation: PriviledgedActionException
>> as:Myuser cause:java.util.concurrent.TimeoutException: Futures timed out
>> after [30 seconds]
>> Exception in thread "main"
>> java.lang.reflect.UndeclaredThrowableException: Unknown exception in doAs
>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1134)
>>     at
>> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:52)
>>     at
>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:113)
>>     at
>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:156)
>>     at
>> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>> Caused by: java.security.PrivilegedActionException:
>> java.util.concurrent.TimeoutException: Futures timed out after [30 seconds]
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:415)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>>     ... 4 more
>> Caused by: java.util.concurrent.TimeoutException: Futures timed out after
>> [30 seconds]
>>     at
>> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
>>     at
>> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
>>     at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
>>     at
>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>     at scala.concurrent.Await$.result(package.scala:107)
>>     at
>> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:125)
>>     at
>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:53)
>>     at
>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:52)
>>     ... 7 more
>>
>
> Any ideas?
>
> Thanks.
>
> On Tue, Nov 4, 2014 at 11:29 AM, Akhil Das <akhil@sigmoidanalytics.com>
> wrote:
>
>> If you want to run the spark application from a remote machine, then you
>> have to at least set the following configurations properly.
>>
>> *spark.driver.host* - points to the ip/host from where you are
>> submitting the job (make sure you are able to ping this from the cluster)
>>
>> *spark.driver.port* - set it to a port number which is accessible from
>> the spark cluster.
>>
>> You can look at more configuration options over here.
>> <http://spark.apache.org/docs/latest/configuration.html#networking>
>>
>> Thanks
>> Best Regards
>>
>> On Tue, Nov 4, 2014 at 6:07 AM, Saiph Kappa <saiph.kappa@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am trying to submit a job to a spark cluster running on a single
>>> machine (1 master + 1 worker) with hadoop 1.0.4. I submit it in the code:
>>> «val sparkConf = new
>>> SparkConf().setMaster("spark://myserver:7077").setAppName("MyApp").setJars(Array("target/my-app-1.0-SNAPSHOT.jar"))».
>>>
>>> When I run this application on the same machine as the cluster
>>> everything works fine.
>>>
>>> But when I run it from a remote machine I get the following error:
>>>
>>> Using Spark's default log4j profile:
>>>> org/apache/spark/log4j-defaults.properties
>>>> 14/11/04 00:15:38 INFO CoarseGrainedExecutorBackend: Registered signal
>>>> handlers for [TERM, HUP, INT]
>>>> 14/11/04 00:15:38 INFO SecurityManager: Changing view acls to:
>>>> myuser,Myuser
>>>> 14/11/04 00:15:38 INFO SecurityManager: Changing modify acls to:
>>>> myuser,Myuser
>>>> 14/11/04 00:15:38 INFO SecurityManager: SecurityManager: authentication
>>>> disabled; ui acls disabled; users with view permissions: Set(myuser,
>>>> Myuser); users with modify permissions: Set(myuser, Myuser)
>>>> 14/11/04 00:15:38 INFO Slf4jLogger: Slf4jLogger started
>>>> 14/11/04 00:15:38 INFO Remoting: Starting remoting
>>>> 14/11/04 00:15:38 INFO Remoting: Remoting started; listening on
>>>> addresses :[akka.tcp://driverPropsFetcher@myserver:49190]
>>>> 14/11/04 00:15:38 INFO Remoting: Remoting now listens on addresses:
>>>> [akka.tcp://driverPropsFetcher@myserver:49190]
>>>> 14/11/04 00:15:38 INFO Utils: Successfully started service
>>>> 'driverPropsFetcher' on port 49190.
>>>> 14/11/04 00:15:38 WARN Remoting: Tried to associate with unreachable
>>>> remote address [akka.tcp://sparkDriver@mylaptop:57418]. Address is now
>>>> gated for 60000 ms, all messages to this address will be delivered to dead
>>>> letters.
>>>> 14/11/04 00:16:08 ERROR UserGroupInformation:
>>>> PriviledgedActionException as:Myuser
>>>> cause:java.util.concurrent.TimeoutException: Futures timed out after [30
>>>> seconds]
>>>> Exception in thread "main"
>>>> java.lang.reflect.UndeclaredThrowableException: Unknown exception in doAs
>>>>     at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1134)
>>>>     at
>>>> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:52)
>>>>     at
>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:113)
>>>>     at
>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:156)
>>>>     at
>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>>> Caused by: java.security.PrivilegedActionException:
>>>> java.util.concurrent.TimeoutException: Futures timed out after [30 seconds]
>>>>     at java.security.AccessController.doPrivileged(Native Method)
>>>>     at javax.security.auth.Subject.doAs(Subject.java:415)
>>>>     at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>>>>     ... 4 more
>>>> Caused by: java.util.concurrent.TimeoutException: Futures timed out
>>>> after [30 seconds]
>>>>     at
>>>> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
>>>>     at
>>>> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
>>>>     at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
>>>>     at
>>>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>>>     at scala.concurrent.Await$.result(package.scala:107)
>>>>     at
>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:125)
>>>>     at
>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:53)
>>>>     at
>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:52)
>>>>     ... 7 more
>>>>
>>>
>>> I know this has something to do with hadoop permissions. I have checked
>>> and all necessary hadoop ports in the server are open and accessible from
>>> outside.
>>>
>>> How can I configure the right permissions?
>>>
>>> Thanks.
>>>
>>
>>
>

Mime
View raw message