spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arush Kharbanda <ar...@sigmoidanalytics.com>
Subject Re: Error when running spark in debug mode
Date Sat, 31 Jan 2015 08:36:25 GMT
Can you share your log4j file.

On Sat, Jan 31, 2015 at 1:35 PM, Arush Kharbanda <arush@sigmoidanalytics.com
> wrote:

> Hi Ankur,
>
> Its running fine for me for spark 1.1 and changes to log4j properties
> file.
>
> Thanks
> Arush
>
> On Fri, Jan 30, 2015 at 9:49 PM, Ankur Srivastava <
> ankur.srivastava@gmail.com> wrote:
>
>> Hi Arush
>>
>> I have configured log4j by updating the file log4j.properties in
>> SPARK_HOME/conf folder.
>>
>> If it was a log4j defect we would get error in debug mode in all apps.
>>
>> Thanks
>> Ankur
>>  Hi Ankur,
>>
>> How are you enabling the debug level of logs. It should be a log4j
>> configuration. Even if there would be some issue it would be in log4j and
>> not in spark.
>>
>> Thanks
>> Arush
>>
>> On Fri, Jan 30, 2015 at 4:24 AM, Ankur Srivastava <
>> ankur.srivastava@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> When ever I enable DEBUG level logs for my spark cluster, on running a
>>> job all the executors die with the below exception. On disabling the DEBUG
>>> logs my jobs move to the next step.
>>>
>>>
>>> I am on spark-1.1.0
>>>
>>> Is this a known issue with spark?
>>>
>>> Thanks
>>> Ankur
>>>
>>> 2015-01-29 22:27:42,467 [main] INFO  org.apache.spark.SecurityManager -
>>> SecurityManager: authentication disabled; ui acls disabled; users with view
>>> permissions: Set(ubuntu); users with modify permissions: Set(ubuntu)
>>>
>>> 2015-01-29 22:27:42,478 [main] DEBUG org.apache.spark.util.AkkaUtils -
>>> In createActorSystem, requireCookie is: off
>>>
>>> 2015-01-29 22:27:42,871
>>> [driverPropsFetcher-akka.actor.default-dispatcher-4] INFO
>>> akka.event.slf4j.Slf4jLogger - Slf4jLogger started
>>>
>>> 2015-01-29 22:27:42,912
>>> [driverPropsFetcher-akka.actor.default-dispatcher-4] INFO  Remoting -
>>> Starting remoting
>>>
>>> 2015-01-29 22:27:43,057
>>> [driverPropsFetcher-akka.actor.default-dispatcher-4] INFO  Remoting -
>>> Remoting started; listening on addresses :[akka.tcp://
>>> driverPropsFetcher@10.77.9.155:36035]
>>>
>>> 2015-01-29 22:27:43,060
>>> [driverPropsFetcher-akka.actor.default-dispatcher-4] INFO  Remoting -
>>> Remoting now listens on addresses: [akka.tcp://
>>> driverPropsFetcher@10.77.9.155:36035]
>>>
>>> 2015-01-29 22:27:43,067 [main] INFO  org.apache.spark.util.Utils -
>>> Successfully started service 'driverPropsFetcher' on port 36035.
>>>
>>> 2015-01-29 22:28:13,077 [main] ERROR
>>> org.apache.hadoop.security.UserGroupInformation -
>>> PriviledgedActionException as:ubuntu
>>> cause:java.util.concurrent.TimeoutException: Futures timed out after [30
>>> seconds]
>>>
>>> Exception in thread "main"
>>> java.lang.reflect.UndeclaredThrowableException: Unknown exception in doAs
>>>
>>>         at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1134)
>>>
>>>         at
>>> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:52)
>>>
>>>         at
>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:113)
>>>
>>>         at
>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:156)
>>>
>>>         at
>>> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
>>>
>>> Caused by: java.security.PrivilegedActionException:
>>> java.util.concurrent.TimeoutException: Futures timed out after [30 seconds]
>>>
>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>
>>>         at javax.security.auth.Subject.doAs(Subject.java:415)
>>>
>>>         at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>>>
>>>         ... 4 more
>>>
>>> Caused by: java.util.concurrent.TimeoutException: Futures timed out
>>> after [30 seconds]
>>>
>>>         at
>>> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
>>>
>>>         at
>>> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
>>>
>>>         at
>>> scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
>>>
>>>         at
>>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>>>
>>>         at scala.concurrent.Await$.result(package.scala:107)
>>>
>>>         at
>>> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:125)
>>>
>>>         at
>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:53)
>>>
>>>         at
>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:52)
>>>
>>>         ... 7 more
>>>
>>
>>
>>
>> --
>>
>> [image: Sigmoid Analytics] <http://htmlsig.com/www.sigmoidanalytics.com>
>>
>> *Arush Kharbanda* || Technical Teamlead
>>
>> arush@sigmoidanalytics.com || www.sigmoidanalytics.com
>>
>
>
>
> --
>
> [image: Sigmoid Analytics] <http://htmlsig.com/www.sigmoidanalytics.com>
>
> *Arush Kharbanda* || Technical Teamlead
>
> arush@sigmoidanalytics.com || www.sigmoidanalytics.com
>



-- 

[image: Sigmoid Analytics] <http://htmlsig.com/www.sigmoidanalytics.com>

*Arush Kharbanda* || Technical Teamlead

arush@sigmoidanalytics.com || www.sigmoidanalytics.com

Mime
View raw message