Hi Ankur,

Its running fine for me for spark 1.1 and changes to log4j properties file. 

Thanks
Arush

On Fri, Jan 30, 2015 at 9:49 PM, Ankur Srivastava <ankur.srivastava@gmail.com> wrote:

Hi Arush

I have configured log4j by updating the file log4j.properties in SPARK_HOME/conf folder.

If it was a log4j defect we would get error in debug mode in all apps.

Thanks
Ankur

Hi Ankur,

How are you enabling the debug level of logs. It should be a log4j configuration. Even if there would be some issue it would be in log4j and not in spark.

Thanks
Arush

On Fri, Jan 30, 2015 at 4:24 AM, Ankur Srivastava <ankur.srivastava@gmail.com> wrote:
Hi,

When ever I enable DEBUG level logs for my spark cluster, on running a job all the executors die with the below exception. On disabling the DEBUG logs my jobs move to the next step.


I am on spark-1.1.0

Is this a known issue with spark?

Thanks
Ankur

2015-01-29 22:27:42,467 [main] INFO  org.apache.spark.SecurityManager - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ubuntu); users with modify permissions: Set(ubuntu)

2015-01-29 22:27:42,478 [main] DEBUG org.apache.spark.util.AkkaUtils - In createActorSystem, requireCookie is: off

2015-01-29 22:27:42,871 [driverPropsFetcher-akka.actor.default-dispatcher-4] INFO  akka.event.slf4j.Slf4jLogger - Slf4jLogger started

2015-01-29 22:27:42,912 [driverPropsFetcher-akka.actor.default-dispatcher-4] INFO  Remoting - Starting remoting

2015-01-29 22:27:43,057 [driverPropsFetcher-akka.actor.default-dispatcher-4] INFO  Remoting - Remoting started; listening on addresses :[akka.tcp://driverPropsFetcher@10.77.9.155:36035]

2015-01-29 22:27:43,060 [driverPropsFetcher-akka.actor.default-dispatcher-4] INFO  Remoting - Remoting now listens on addresses: [akka.tcp://driverPropsFetcher@10.77.9.155:36035]

2015-01-29 22:27:43,067 [main] INFO  org.apache.spark.util.Utils - Successfully started service 'driverPropsFetcher' on port 36035.

2015-01-29 22:28:13,077 [main] ERROR org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException as:ubuntu cause:java.util.concurrent.TimeoutException: Futures timed out after [30 seconds]

Exception in thread "main" java.lang.reflect.UndeclaredThrowableException: Unknown exception in doAs

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1134)

        at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:52)

        at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:113)

        at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:156)

        at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)

Caused by: java.security.PrivilegedActionException: java.util.concurrent.TimeoutException: Futures timed out after [30 seconds]

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)

        ... 4 more

Caused by: java.util.concurrent.TimeoutException: Futures timed out after [30 seconds]

        at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)

        at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)

        at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)

        at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)

        at scala.concurrent.Await$.result(package.scala:107)

        at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:125)

        at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:53)

        at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:52)

        ... 7 more




--

Sigmoid Analytics

Arush Kharbanda || Technical Teamlead

arush@sigmoidanalytics.com || www.sigmoidanalytics.com




--

Sigmoid Analytics

Arush Kharbanda || Technical Teamlead

arush@sigmoidanalytics.com || www.sigmoidanalytics.com