spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles <charles...@cenx.com>
Subject Deadlock between spark logging and wildfly logging
Date Fri, 28 Nov 2014 16:01:47 GMT
We create spark context in an application running inside wildfly container.
When spark context is created, we see following entires in the wildfly log.
After the log4j-default.properties is loaded, every entry from spark is
printed out twice. And after running for a while, we start to see deadlock
between spark logging thread and wildfly logging thread. 

 Can I control the spark logging in the driver application? How can I turn
it off in the driver application? How can I control the level of spark logs
in the driver application?

2014-11-27 14:39:26,719 INFO  [akka.event.slf4j.Slf4jLogger]
(spark-akka.actor.default-dispatcher-4) Slf4jLogger started
2014-11-27 14:39:26,917 INFO  [Remoting]
(spark-akka.actor.default-dispatcher-2) Starting remoting
2014-11-27 14:39:27,719 INFO  [Remoting]
(spark-akka.actor.default-dispatcher-2) Remoting started; listening on
addresses :[akka.tcp://spark@172.32.1.12:43918]
2014-11-27 14:39:27,733 INFO  [Remoting]
(spark-akka.actor.default-dispatcher-2) Remoting now listens on addresses:
[akka.tcp://spark@172.32.1.12:43918]
2014-11-27 14:39:27,892 INFO  [org.apache.spark.SparkEnv] (MSC service
thread 1-16) Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
2014-11-27 14:39:27,895 ERROR [stderr] (MSC service thread 1-16) 14/11/27
14:39:27 INFO SparkEnv: Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
2014-11-27 14:39:27,896 INFO  [org.apache.spark.SparkEnv] (MSC service
thread 1-16) Registering BlockManagerMaster
2014-11-27 14:39:27,896 ERROR [stderr] (MSC service thread 1-16) 14/11/27
14:39:27 INFO SparkEnv: Registering BlockManagerMaster
2014-11-27 14:39:28,041 INFO  [org.apache.spark.storage.DiskBlockManager]
(MSC service thread 1-16) Created local directory at
/tmp/spark-local-20141127143928-d33c
2014-11-27 14:39:28,041 ERROR [stderr] (MSC service thread 1-16) 14/11/27
14:39:28 INFO DiskBlockManager: Created local directory at
/tmp/spark-local-20141127143928-d33c
2014-11-27 14:39:28,055 INFO  [org.apache.spark.storage.MemoryStore] (MSC
service thread 1-16) MemoryStore started with capacity 4.3 GB.
2014-11-27 14:39:28,055 ERROR [stderr] (MSC service thread 1-16) 14/11/27
14:39:28 INFO MemoryStore: MemoryStore started with capacity 4.3 GB.
2014-11-27 14:39:28,117 INFO  [org.apache.spark.network.ConnectionManager]
(MSC service thread 1-16) Bound socket to port 34018 with id =
ConnectionManagerId(ip-172-32-1-12,34018)
2014-11-27 14:39:28,118 ERROR [stderr] (MSC service thread 1-16) 14/11/27
14:39:28 INFO ConnectionManager: Bound socket to port 34018 with id =
ConnectionManagerId(ip-172-32-1-12,34018)
2014-11-27 14:39:28,162 INFO  [org.apache.spark.storage.BlockManagerMaster]
(MSC service thread 1-16) Trying to register BlockManager
2014-11-27 14:39:28,163 ERROR [stderr] (MSC service thread 1-16) 14/11/27
14:39:28 INFO BlockManagerMaster: Trying to register BlockManager
2014-11-27 14:39:28,181 INFO 
[org.apache.spark.storage.BlockManagerMasterActor$BlockManagerInfo]
(spark-akka.actor.default-dispatcher-3) Registering block manager
ip-172-32-1-12:34018 with 4.3 GB RAM
2014-11-27 14:39:28,185 ERROR [stderr]
(spark-akka.actor.default-dispatcher-3) 14/11/27 14:39:28 INFO
BlockManagerMasterActor$BlockManagerInfo: Registering block manager
ip-172-32-1-12:34018 with 4.3 GB RAM



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Deadlock-between-spark-logging-and-wildfly-logging-tp20009.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message